For a machine learning task I have coded with support of the internet a configurable PoinNet Version in tensorflow keras.
The keras example from PointNet I got here: Point cloud classification with PointNet
I am using tensorflow = "2.11.0"
What do I want?
- Implemented an easy version of PointNet (see code below)
- I have some stl files which I load with
open3d = "^0.18.0"to convert it to PointClouds - I want to use
tf.data.Datasetpipe to load the data and pre- and postprocess it - I want to predict a certain curve/values on the end, depending on the input (but this is not the problem)
- For simplicity and clarity I implemented two options: 1. Load all pointcloud data in one numpy array before training AND 2.Load pointcloud data while training via
tf.py_function
My Problem
- When using option 2 (load pointcloud data while training via
tf.py_function): regardless of the number of points in a point cloud, the data is passed through the neural network, although it does not match theINPUT_SHAPEfrom the neural net - When I use option 1 (load all pointclouds before training and then build the
tf.data.Datasetpipe): everything works as expected. If the pointcloud shape does not match theINPUT_SHAPEof the neural net, an error occurs. - Why is there the strange behavior? Is it because of the usage of
tf.py_function?
My code block for the PointNet
import os import numpy as np import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers import open3d as o3d import pandas as pd import random ############################################################################################## # define constants NUM_POINTS = 20 INPUT_SHAPE = [20, 3] BATCHSIZE = 6 NUM_CLASSES = 10 NUM_SAMPLES = 6 WHICH_LOADING = "all" # all, single_py_function PATH_FOLDER = "data/few_examples/stl_examples/" ############################################################################################## # define loading functions def load_all_pointclouds(list_filepaths): list_arrays = [] for path in list_filepaths: mesh = o3d.io.read_triangle_mesh(path) pointcloud = mesh.sample_points_poisson_disk( number_of_points=NUM_POINTS, init_factor=5 ) array_pointcloud = np.asarray(pointcloud.points) list_arrays.append(array_pointcloud) print("array_pointcloud.shape: ", array_pointcloud.shape) arrays_pointcloud = np.array(list_arrays) return arrays_pointcloud def load_single_pointcloud(filepath): filepath = filepath.numpy().decode("utf-8") mesh = o3d.io.read_triangle_mesh(filepath) pointcloud = mesh.sample_points_poisson_disk( number_of_points=NUM_POINTS, init_factor=5 ) array_pointcloud = np.asarray(pointcloud.points) array_pointcloud = np.float32(array_pointcloud) array_pointcloud = tf.ensure_shape(array_pointcloud, INPUT_SHAPE) print("array_pointcloud.shape: ", array_pointcloud.shape) return array_pointcloud ############################################################################################## ### create filepaths list_files = os.listdir(PATH_FOLDER) list_filepaths = [PATH_FOLDER + x for x in list_files] ### create y data df_y = pd.DataFrame(np.random.rand(NUM_SAMPLES, NUM_CLASSES)) ds_y = tf.data.Dataset.from_tensor_slices(df_y) ### logic for loading while train or preload all data if WHICH_LOADING == "single_py_function": df_3d = pd.DataFrame(list_filepaths) ds_3d = tf.data.Dataset.from_tensor_slices(df_3d) ds_3d = ds_3d.map( lambda filepath: tf.py_function( load_single_pointcloud, [filepath[0]], Tout=tf.float32 ) ) if WHICH_LOADING == "all": arr_3d = load_all_pointclouds(list_filepaths) ds_3d = tf.data.Dataset.from_tensor_slices(arr_3d) ### zip datasets and postprocess data ds = tf.data.Dataset.zip((ds_3d, ds_y)) ds = ds.shuffle(buffer_size=1000, seed=42).batch(BATCHSIZE).prefetch(tf.data.AUTOTUNE) ### check ds data for i, element in enumerate(ds.as_numpy_iterator()): print("array_pointcloud.type: ", type(element[0])) print("array_pointcloud.shape: ", element[0].shape) print("y.shape", element[1].shape) ### Build a model def conv_bn(x, filters): x = layers.Conv1D(filters, kernel_size=1, padding="valid")(x) x = layers.BatchNormalization(momentum=0.0)(x) return layers.Activation("relu")(x) def dense_bn(x, filters): x = layers.Dense(filters)(x) x = layers.BatchNormalization(momentum=0.0)(x) return layers.Activation("relu")(x) class OrthogonalRegularizer(keras.regularizers.Regularizer): def __init__(self, num_features, l2reg=0.001): self.num_features = num_features self.l2reg = l2reg self.eye = tf.eye(num_features) def __call__(self, x): x = tf.reshape(x, (-1, self.num_features, self.num_features)) xxt = tf.tensordot(x, x, axes=(2, 2)) xxt = tf.reshape(xxt, (-1, self.num_features, self.num_features)) return tf.reduce_sum(self.l2reg * tf.square(xxt - self.eye)) def tnet(inputs, num_features): # Initalise bias as the indentity matrix bias = keras.initializers.Constant(np.eye(num_features).flatten()) reg = OrthogonalRegularizer(num_features) x = conv_bn(inputs, 32) x = conv_bn(x, 64) x = conv_bn(x, 512) x = layers.GlobalMaxPooling1D()(x) x = dense_bn(x, 256) x = dense_bn(x, 128) x = layers.Dense( num_features * num_features, kernel_initializer="zeros", bias_initializer=bias, activity_regularizer=reg, )(x) feat_T = layers.Reshape((num_features, num_features))(x) return layers.Dot(axes=(2, 1))([inputs, feat_T]) inputs = keras.Input(shape=INPUT_SHAPE) x = tnet(inputs, 3) x = conv_bn(x, 32) x = conv_bn(x, 32) x = tnet(x, 32) x = conv_bn(x, 32) x = conv_bn(x, 64) x = conv_bn(x, 512) x = layers.GlobalMaxPooling1D()(x) x = dense_bn(x, 256) x = layers.Dropout(0.3)(x) x = dense_bn(x, 128) x = layers.Dropout(0.3)(x) outputs = layers.Dense(NUM_CLASSES, activation="softmax")(x) model = keras.Model(inputs=inputs, outputs=outputs, name="pointnet") ### compile and fit data # model.summary() model.compile( loss="mse", optimizer=keras.optimizers.Adam(learning_rate=0.001), ) model.fit(ds, epochs=2) Print Results on different configurations
Configuration 1 - loading all before train works fine when shapes match
NUM_POINTS = 20 INPUT_SHAPE = [20, 3] BATCHSIZE = 6 WHICH_LOADING = "all_before" Print Result for Config 1
array_pointcloud.shape in load_all_pointclouds function: (20, 3) array_pointcloud.shape in load_all_pointclouds function: (20, 3) array_pointcloud.shape in load_all_pointclouds function: (20, 3) array_pointcloud.shape in load_all_pointclouds function: (20, 3) array_pointcloud.shape in load_all_pointclouds function: (20, 3) array_pointcloud.shape in load_all_pointclouds function: (20, 3) array_pointcloud.type in dataset loop: <class 'numpy.ndarray'> array_pointcloud.shape in dataset loop: (6, 20, 3) y.shape in dataset loop (6, 10) INPUT_SHAPE neural net: [20, 3] NUM_POINTS predefined: 20 BATCHSIZE predefined: 6 Epoch 1/2 1/1 [==============================] - 9s 9s/step - loss: 0.7237 Epoch 2/2 1/1 [==============================] - 0s 16ms/step - loss: 0.6924 Configuration 2 - loading all before train throws error as expected when shapes do not match
NUM_POINTS = 21 INPUT_SHAPE = [20, 3] BATCHSIZE = 6 WHICH_LOADING = "all_before" Print Result for Config 2
array_pointcloud.shape in load_all_pointclouds function: (21, 3) array_pointcloud.shape in load_all_pointclouds function: (21, 3) array_pointcloud.shape in load_all_pointclouds function: (21, 3) array_pointcloud.shape in load_all_pointclouds function: (21, 3) array_pointcloud.shape in load_all_pointclouds function: (21, 3) array_pointcloud.shape in load_all_pointclouds function: (21, 3) array_pointcloud.type in dataset loop: <class 'numpy.ndarray'> array_pointcloud.shape in dataset loop: (6, 21, 3) y.shape in dataset loop (6, 10) INPUT_SHAPE neural net: [20, 3] NUM_POINTS predefined: 21 BATCHSIZE predefined: 6 Error message for Config 2
ValueError: Input 0 of layer "pointnet" is incompatible with the layer: expected shape=(None, 20, 3), found shape=(None, 21, 3) Configuration 3 - loading while train works fine when shapes match
NUM_POINTS = 20 INPUT_SHAPE = [20, 3] BATCHSIZE = 6 WHICH_LOADING = "single_py_function_while_train" Print Result for Config 3
array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.type in dataset loop: <class 'numpy.ndarray'> array_pointcloud.shape in dataset loop: (6, 20, 3) y.shape in dataset loop (6, 10) INPUT_SHAPE neural net: [20, 3] NUM_POINTS predefined: 20 BATCHSIZE predefined: 6 Epoch 1/2 array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) 1/1 [==============================] - 16s 16s/step - loss: 0.5778 Epoch 2/2 array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) array_pointcloud.shape in load_single_pointcloud function: (20, 3) 1/1 [==============================] - 0s 193ms/step - loss: 0.5620 Configuration 4 - loading while train does not work as expected if shapes do not match. It should throw an error. But it does not.
NUM_POINTS = 21 INPUT_SHAPE = [20, 3] BATCHSIZE = 6 WHICH_LOADING = "single_py_function_while_train" Print Result for Config 3
array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.type in dataset loop: <class 'numpy.ndarray'> array_pointcloud.shape in dataset loop: (6, 21, 3) y.shape in dataset loop (6, 10) INPUT_SHAPE neural net: [20, 3] NUM_POINTS predefined: 21 BATCHSIZE predefined: 6 Epoch 1/2 array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) 1/1 [==============================] - 13s 13s/step - loss: 0.6657 Epoch 2/2 array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) array_pointcloud.shape in load_single_pointcloud function: (21, 3) 1/1 [==============================] - 0s 64ms/step - loss: 0.6431 Conclusion
So, how is it possible that a tf neural net can be successful trained, although the data shape and the input shape does not match when loading and feeding data while train? Is it because of the usage of tf.py_function?
Thanks for help.