Yocto 1.7: problems when adding Python3 modules (paramiko, pycrypt, distribute)

[lukaszzemla]

Hi!

I'd like to ad to my image Python 3 modules, especially python3-paramiko and required by it module pycrypto (I'd like to use P1020RDB-PC with Python 3 installed as some SFTP/SSH testbench station). I use freescale_sdk_ubuntu14.04_32bit_PPCE500V2_20150108 Virtual Host Environment. I'm a newbie in a Yocto, so maybe I'm doing something wrong. My way of thinking was as following:

There is no paramiko recipe for Python 3, so I created new recipe python3-pycrypto_2.6.bb basing on existing recipe python-pycrypto_2.6.bb (I renamed all occurencies of'distutils' to 'distutils3' - is it the right way to tell Yocto that I want to have Python3 module?).

In a similar way I created python3-paramiko_1.15.2.bb recipe (both are attached). When I try to bitbake pycrypto, I get following error:

NOTE: Preparing runqueue

NOTE: Executing SetScene Tasks

NOTE: Executing RunQueue Tasks

ERROR: Function failed: do_compile (log file is located at /home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/work/ppce500v2-fsl-linux-gnuspe/python3-pycrypto/2.6-r0/temp/log.do_compile.25789)

ERROR: Logfile of failure stored in: /home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/work/ppce500v2-fsl-linux-gnuspe/python3-pycrypto/2.6-r0/temp/log.do_compile.25789

Log data follows:

| DEBUG: SITE files ['endian-big', 'bit-32', 'powerpc-common', 'common-linux', 'common-glibc', 'powerpc-linux', 'powerpc32-linux', 'powerpc-linux-gnuspe', 'common']

| DEBUG: Executing shell function do_compile

| Traceback (most recent call last):

|   File "setup.py", line 45, in <module>

|     import struct

|   File "/home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/sysroots/i686-linux/usr/lib/python3.3/struct.py", line 12, in <module>

|     from _struct import *

| ImportError: No module named '_struct'

| ERROR: python3 setup.py build_ext execution failed.

| WARNING: exit code 1 from a shell command.

| ERROR: Function failed: do_compile (log file is located at /home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/work/ppce500v2-fsl-linux-gnuspe/python3-pycrypto/2.6-r0/temp/log.do_compile.25789)

ERROR: Task 580 (/home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/meta-woodward/recipes-devtools/python3-pycrypto/python3-pycrypto_2.6.bb, do_compile) failed with exit code '1'

NOTE: Tasks Summary: Attempted 1425 tasks of which 1424 didn't need to be rerun and 1 failed.

No currently running tasks (1424 of 2422)

Summary: 1 task failed:

  /home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/meta-woodward/recipes-devtools/python3-pycrypto/python3-pycrypto_2.6.bb, do_compile

Summary: There was 1 WARNING message shown.

Summary: There was 1 ERROR message shown, returning a non-zero exit code.

freescale@freescale-sdk:~/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release$

I noticed, that I also get errors when building python3-distribute package:

NOTE: Preparing runqueue

NOTE: Executing SetScene Tasks

NOTE: Executing RunQueue Tasks

ERROR: Function failed: do_compile (log file is located at /home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/work/ppce500v2-fsl-linux-gnuspe/python3-distribute/0.6.32-ml5/temp/log.do_compile.26341)

ERROR: Logfile of failure stored in: /home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/work/ppce500v2-fsl-linux-gnuspe/python3-distribute/0.6.32-ml5/temp/log.do_compile.26341

Log data follows:

| DEBUG: Executing shell function do_compile

| Traceback (most recent call last):

|   File "setup.py", line 45, in <module>

|     exec(init_file.read(), d)

|   File "<string>", line 8, in <module>

|   File "/home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/work/ppce500v2-fsl-linux-gnuspe/python3-distribute/0.6.32-ml5/distribute-0.6.32/build/src/setuptools/__init__.py", line 2, in <module>

|     from setuptools.extension import Extension, Library

|   File "/home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/work/ppce500v2-fsl-linux-gnuspe/python3-distribute/0.6.32-ml5/distribute-0.6.32/build/src/setuptools/extension.py", line 5, in <module>

|     from setuptools.dist import _get_unpatched

|   File "/home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/work/ppce500v2-fsl-linux-gnuspe/python3-distribute/0.6.32-ml5/distribute-0.6.32/build/src/setuptools/dist.py", line 103

|     except ValueError, e:

|                      ^

| SyntaxError: invalid syntax

| ERROR: python3 setup.py build_ext execution failed.

| WARNING: exit code 1 from a shell command.

| ERROR: Function failed: do_compile (log file is located at /home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/work/ppce500v2-fsl-linux-gnuspe/python3-distribute/0.6.32-ml5/temp/log.do_compile.26341)

ERROR: Task 6 (/home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/poky/meta/recipes-devtools/python/python3-distribute_0.6.32.bb, do_compile) failed with exit code '1'

NOTE: Tasks Summary: Attempted 584 tasks of which 583 didn't need to be rerun and 1 failed.

No currently running tasks (583 of 634)

Summary: 1 task failed:

  /home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/poky/meta/recipes-devtools/python/python3-distribute_0.6.32.bb, do_compile

Summary: There was 1 WARNING message shown.

Summary: There was 1 ERROR message shown, returning a non-zero exit code.

freescale@freescale-sdk:~/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release$

For me it looks like broken Python environment used during builds... Any ideas?

Thank you in advance.

Best regards,

Lukasz

Original Attachment has been moved to: python3-pycrypto_2.6.bb.zip

Original Attachment has been moved to: python3-paramiko_1.15.2.bb.zip

[yipingwang]

Hello Lukasz Zemla,

According to your error log, it looks that the file  in "/home/freescale/SDK/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release/tmp/sysroots/i686-linux/usr/lib/python3.3/struct.py" is abnormal.

I will download Ubuntu 14.04 virtual machine ISO, and do more investigation.

Thanks,

Yiping

[yipingwang]

Hello Lukasz Zemla,

Please put the attached recipe in meta-cloud-services/meta-openstack/recipes-devtools/python/, I verified the command "bitbake python3-paramiko" without problem in my SDK 1.7 Yocto environment.

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[lukaszzemla]

I did a test on clean Yocto SDK 1.7 installation using your recipes... Unfortunately it did not work... I do not know what are differences...

Attached log.

[yipingwang]

Hello Lukasz Zemla,

Probably this problem is related with Ubuntu environment.

After extract the recipe folder, execute "cp -r recipe/*  meta-cloud-services/meta-openstack/recipes-devtools/python/" , then use "bitbake python3-paramiko", this package is built without any problem.

If your problem remains, would you please send this file build_p1020rdb_release/tmp/work/ppce500v2-fsl-linux-gnuspe/python3-pycrypto/2.6-r0/temp/log.do_compile.25789* to me?

Have a great day,
Yiping

-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[lukaszzemla]

Hello Yiping Wang,

What host Linux distribution and version do you use?

I did some tests last days - and in fact, the problem is with Ubuntu 14.04 environment.

I've tried on 2 different Ubuntu 14.04 virtual machines (created from scratch) - it did not work.

I've tried on freescale_sdk_ubuntu14.04_32bit_PPCE500V2_20150108 Virtual Host Environment image - it did not work.

It works fine on Ubuntu 12.04.05. I'm going to use that version for my purposes.

The open question is why it does not work in Ubuntu 14.04... It is latest Long Term Support version, so it would be nice to have it fully operational. As I wrote - I can use 12.04 now and case is closed for me, however I'd appreciate if you (and/or development team) could investigate that more.

Thank you very much for your support. Have a great day!

[lukaszzemla]

Hello Yiping Wang,

Unfortunately the problem still exists :-(. I did all steps like in attached steps_done.txt file.

File requested by you (log.do_compile.25789) is related with my first post, where I used my recipes (I assume they're bad). When I use your correct recipes, the error is different - it is in python3-distribute-native. Anyway - I'm attaching tmp/work/ppce500v2-fsl-linux-gnuspe/python* and /tmp/work/i686-linux/python3-distribute-native/0.6.32-ml5/temp/* folders.

Error initially appeared on Virtual Machine from Freescale webpage (freescale_sdk_ubuntu14.04_32bit_PPCE500V2_20150108: I converted vdi file to vmdk - it shouldn't break anything). My last 2 attempts come from Virtual Machine with Ubuntu 14.04-01 (default install process, no manipulations on python packages)

ertsuser@VM-Ubuntu-SDK:~/clean_sdk17/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release$ uname -a

Linux VM-Ubuntu-SDK 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:45:15 UTC 2015 i686 i686 i686 GNU/Linux

ertsuser@VM-Ubuntu-SDK:~/clean_sdk17/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release$ python -V

Python 2.7.6

ertsuser@VM-Ubuntu-SDK:~/clean_sdk17/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release$ python2 -V

Python 2.7.6

ertsuser@VM-Ubuntu-SDK:~/clean_sdk17/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release$ python3 -V

Python 3.4.0

ertsuser@VM-Ubuntu-SDK:~/clean_sdk17/QorIQ-SDK-V1.7-20141218-yocto/build_p1020rdb_release$

Have a nice day,

Lukasz