Anomaly Detection - Cannot run adt_train.py for training my own model

[Jodor]

Hi,

I have the MIMXRT1060-EVK and I'm trying to train my own model of anomaly detection with evkmimxrt1060_agm01_tensorflow_lite_adt example but I can't.

I tried to install the python libraries that Appendix A gives but some of them were obsolete so I updated them. Now these libraries are with these versions:

-gast 0.3.3

-google-pasta 0.1.8

-grpcio 1.32.0

-keras-preprocessing 1.1.1

-matplotlib 3.3.0

-numpy 1.17.5

-pandas 1.1.1

-scikit-learn 0.23.2

-scipy 1.5.2

-six 1.15.0

-tensorboard 2.3.0

-tensorflow 2.3.1

-tensorflow-estimator 2.3.0

With these updates, I can't run the adt_train.py script. Could someone tell me if libraries are rigth for running adt_train.py file? 

Regards,

Jorge

[jeremyzhou]

Hi Jorge,
Thank you for your interest in NXP Semiconductor products and for the opportunity to serve you.
Before answering your question, I was wondering if you can introduce what the command used to run the adt_train.py, and the error messages after running the adt_train.py.
Looking forward to your reply.
Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Jodor]

Hi @jeremyzhou,

The command that I have used is:

python adt_train.py --input_data="input_data.txt" --validation_data="validation_data.txt" --vizualize=1

and the error I get is:

File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\joblib\externals\cloudpickle\cloudpickle.py", line 133, in _make_cell_set_template_code
return types.CodeType(
TypeError: an integer is required (got type bytes)

I have tried to update de cloudpickle library but it's updated already.

[jeremyzhou]

Hi,
Thanks for your reply.
Did you provide your own input data and validation_data in advance of running the command? If yes, can you upload these data? Then I can give a test on my side.
Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Jodor]

hi,

I got my own input_data and validation_data but doesn't work. I attach you the files.

By the way, do you know where can I get the input and validation data of the example?

Thanks for the help.

[jeremyzhou]

Hi,
Thanks for your reply.
Sorry for the reply late, I'm working on it now, and I also encountered some errors when running the adt_train.py, as my environment is based on TensorFlow 2.0 and Python 3.8.
And after contacting with the SDK team, they'd like to suggest the developers run the python code in the Jupyter Note if the adt_train.py doesn't work normally.

Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Jodor]

Hi,

Thank you again for the help. I'm trying to ejecute it with Jupyter notebook but neither can't. I've got the same problem. Here what it says:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-73edc048c06b> in <module>
----> 1 from sklearn.model_selection import train_test_split

~\Anaconda3\envs\adt\lib\site-packages\sklearn\__init__.py in <module>
     78     from . import _distributor_init  # noqa: F401
     79     from . import __check_build  # noqa: F401
---> 80     from .base import clone
     81     from .utils._show_versions import show_versions
     82 

~\Anaconda3\envs\adt\lib\site-packages\sklearn\base.py in <module>
     19 from . import __version__
     20 from ._config import get_config
---> 21 from .utils import _IS_32BIT
     22 from .utils.validation import check_X_y
     23 from .utils.validation import check_array

~\Anaconda3\envs\adt\lib\site-packages\sklearn\utils\__init__.py in <module>
     21 
     22 from .murmurhash import murmurhash3_32
---> 23 from .class_weight import compute_class_weight, compute_sample_weight
     24 from . import _joblib
     25 from ..exceptions import DataConversionWarning

~\Anaconda3\envs\adt\lib\site-packages\sklearn\utils\class_weight.py in <module>
      5 import numpy as np
      6 
----> 7 from .validation import _deprecate_positional_args
      8 
      9 

~\Anaconda3\envs\adt\lib\site-packages\sklearn\utils\validation.py in <module>
     19 
     20 from numpy.core.numeric import ComplexWarning
---> 21 import joblib
     22 
     23 from contextlib import suppress

~\Anaconda3\envs\adt\lib\site-packages\joblib\__init__.py in <module>
    117 from .numpy_pickle import load
    118 from .compressor import register_compressor
--> 119 from .parallel import Parallel
    120 from .parallel import delayed
    121 from .parallel import cpu_count

~\Anaconda3\envs\adt\lib\site-packages\joblib\parallel.py in <module>
     26 from .my_exceptions import TransportableException
     27 from .disk import memstr_to_bytes
---> 28 from ._parallel_backends import (FallbackToBackend, MultiprocessingBackend,     29                                  ThreadingBackend, SequentialBackend,
     30                                  LokyBackend)

~\Anaconda3\envs\adt\lib\site-packages\joblib\_parallel_backends.py in <module>
     20     from .pool import MemmappingPool
     21     from multiprocessing.pool import ThreadPool
---> 22     from .executor import get_memmapping_executor
     23 
     24     # Compat between concurrent.futures and multiprocessing TimeoutError

~\Anaconda3\envs\adt\lib\site-packages\joblib\executor.py in <module>
     12 from .disk import delete_folder
     13 from ._memmapping_reducer import get_memmapping_reducers
---> 14 from .externals.loky.reusable_executor import get_reusable_executor
     15 
     16 

~\Anaconda3\envs\adt\lib\site-packages\joblib\externals\loky\__init__.py in <module>
     10 
     11 from .backend.context import cpu_count
---> 12 from .backend.reduction import set_loky_pickler
     13 from .reusable_executor import get_reusable_executor
     14 from .cloudpickle_wrapper import wrap_non_picklable_objects

~\Anaconda3\envs\adt\lib\site-packages\joblib\externals\loky\backend\reduction.py in <module>
    123 # global variable to change the pickler behavior
    124 try:
--> 125     from joblib.externals import cloudpickle  # noqa: F401
    126     DEFAULT_ENV = "cloudpickle"
    127 except ImportError:

~\Anaconda3\envs\adt\lib\site-packages\joblib\externals\cloudpickle\__init__.py in <module>
      1 from __future__ import absolute_import
      2 
----> 3 from .cloudpickle import *
      4 
      5 __version__ = '0.8.0'

~\Anaconda3\envs\adt\lib\site-packages\joblib\externals\cloudpickle\cloudpickle.py in <module>
    150 
    151 
--> 152 _cell_set_template_code = _make_cell_set_template_code()
    153 
    154 

~\Anaconda3\envs\adt\lib\site-packages\joblib\externals\cloudpickle\cloudpickle.py in _make_cell_set_template_code()
    131         )
    132     else:
--> 133         return types.CodeType(    134             co.co_argcount,
    135             co.co_kwonlyargcount,

TypeError: an integer is required (got type bytes)

 Regards,

[jeremyzhou]

Hi,
Thanks for your reply.
The error seems to be caused by the import library fails.
I've created a sample code to generate an autoencoder model in the jupyter notebook, definitely, it's just a demo and still needs more work to make it better, however, it can be treated as an entry demo.

# Creates an autoencoder model
import pandas as pd
import numpy as np
​
import sys
import argparse
​
from sklearn.model_selection import train_test_split
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import Sequential, layers
# Prepare dataset
## Preprocess data
def preprocess_data(data):
    #data = (FLAGS.scale_range/3*(data - data.mean())/data.std()).clip(-1,1)
    data = data.diff()[5:]
    data[['ax','ay','az']] /= 10
    #data = (data + 1)/2
    return data


chanels = ['wx', 'wy', 'wz', 'ax', 'ay', 'az']
patch_size = 5
   
# load input data
data = pd.read_csv('input_data.txt', delimiter = ',', dtype = np.int, header=None,names=chanels)
data = preprocess_data(data)

# prepare patches   
samples_nb = int(np.floor(data.shape[0] / float(patch_size))) * patch_size
patch = np.reshape(data[chanels][:samples_nb].values, (-1,len(chanels)*patch_size))

# split into train and test set
train_set, test_set = train_test_split(patch, test_size=0.1, random_state=42)
x_train, x_test = train_set.astype(np.float32) , test_set.astype(np.float32) 
# Build Model
# create autoencoder model
input_dim = test_set.shape[1]   
encoding_dim = 30 

tf.keras.backend.clear_session()

class Autoencoder(keras.Model):

    def __init__(self):
        super(Autoencoder, self).__init__() 

        # Encoders
        self.encoder = Sequential([
            layers.Dense(15, activation=tf.nn.relu),
            layers.Dense(4, activation=tf.nn.relu),
        ])

        # Decoders
        self.decoder = Sequential([
            layers.Dense(4, activation=tf.nn.relu),
            layers.Dense(16, activation=tf.nn.relu),
            layers.Dense(encoding_dim, activation=tf.nn.relu),
        ])


    def call(self, inputs, training=None):
        # [b, 30] => [b, 4]
        h = self.encoder(inputs)
        # [b, 4] => [b, 30]
        x_hat = self.decoder(h)
        return x_hat

model = Autoencoder()
model.build(input_shape=(None, encoding_dim))

optimizer = tf.optimizers.Adamax(lr=lr)

model.compile(optimizer=optimizer,
                    loss='mean_squared_error', 
                    metrics=['accuracy'])

model.summary()

# Train Model
 # set training parameters
nb_epoch = 1000
batch_num = 20

history = model.fit(x_train,x_train,
                    epochs=nb_epoch,
                    batch_size=batch_num,
                    shuffle=True,
                    validation_data=(x_test,x_test),
                    verbose=1)

Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------