TFLite FullyConnected Layers on iMX8MPlus using NNAPI

khuang — Fri, 17 Jun 2022 08:42:39 GMT

We are using the iMX8MPlus module and we are trying to deploy a self-developed tensorflow-lite model on the NPU.

The tf-lite version we are using is the lf-5.15.5_1.0.0 from source.codeaurora.org/external/imx/tensorflow-imx.git We are trying to use the NNAPI delegate to run the model on the NPU, sofar it works for all the convolutional layers, but the FULLY_CONNECTED layers are still having issues.

The error is shows is: Node 0 Operator Builtin Code 9 FULLY_CONNECTED (not delegated) and digging down the source code a bit deeper, we found it was failing in line if (builtin->keep_num_dims) of the file nnapi_delegate.cc:

    case kTfLiteBuiltinFullyConnected: {
      ExpectMaxOpVersion(version, 5, &val_ctx);
      const auto output_type = context->tensors[node->outputs->data[0]].type;
      Expect(output_type != kTfLiteInt16,
             NNAPIValidationFailureType::kUnsupportedOutputType,
             "Unsupported output of type kTfLiteInt16", &val_ctx);
      if (android_sdk_version < kMinSdkVersionForNNAPI12) {
        Expect(!IsHybridOperator(context, builtin_code, node),
               NNAPIValidationFailureType::kUnsupportedHybridOperator,
               "Hybrid operators not supported before NNAPI 1.2", &val_ctx);
        ExpectIsFloatOrUint8Operator(context, node, &val_ctx);
      }
      const auto input_type = context->tensors[node->inputs->data[0]].type;
      if (android_sdk_version < kMinSdkVersionForNNAPI12 &&
          input_type == kTfLiteUInt8) {
        ExpectIsRestrictedScalesCompliant(context, node, &val_ctx);
      }
      auto builtin =
          reinterpret_cast<TfLiteFullyConnectedParams*>(node->builtin_data);
      if (builtin->keep_num_dims) {
        ExpectMinAndroidSdkVersion(android_sdk_version,
                                   kMinSdkVersionForNNAPI13, &val_ctx);
      }
    } break;

So if the flag keep_num_dims is true, then it requires a android sdk version of >= 30, whereas in this version of tf-lite (the NXP customized version) the android_sdk version is defined as 29.

We have been trying to figure out how to set the keep_num_dims flag in the model we are developing, however we cannot find much resource about it. Is there any experience of someone else who has already faced such issue before?

topic TFLite FullyConnected Layers on iMX8MPlus using NNAPI in i.MX Processors

TFLite FullyConnected Layers on iMX8MPlus using NNAPI