Difference between revisions of "Armadeus Integration Test"

From ArmadeusWiki
Jump to: navigation, search
m (developping...)
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
=Armadeus Test Integration=
+
=Armadeus Integration Test=
  
 
Create automatic test & Validation tools   
 
Create automatic test & Validation tools   
Line 5: Line 5:
 
==Changelog==
 
==Changelog==
  
 +
september 2013: Installation of a test server with Hudson / Jenkins
  
 
==Introduction==
 
==Introduction==
Line 23: Line 24:
 
===Compilation Tests===
 
===Compilation Tests===
  
−
Use a build farm to run the tests periodicaly (for each commit). [https://hudson.dev.java.net/ Hudson] or [http://buildbot.net/trac buildbot] are good pretenders. There are many competitors here [http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix CI Matrix] <br>
+
Use a build farm to run the tests periodicaly (for each commit). [https://hudson.dev.java.net/ Hudson] / [http://jenkins-ci.org jenkins] or [http://buildbot.net/trac buildbot] are good pretenders. There are many competitors here [http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix CI Matrix] <br>
−
The packages have to be tested one at a time to avoid dependency problems. '''<- ! not for compilation ?!!'''
+
−
<br>'''Do you mean download packages with make defconfig or test dependencies of each package? '''
+
  
−
#checkout the head of svn
+
#checkout the head of GIT
 
#run build for each supported target  
 
#run build for each supported target  
−
#check build result ?? point 4 is not sufficient ? : '''Yes, It could be. This point aims to check the makefile.'''
+
#check build result  
−
#check presence of binary images  
+
#check presence of binary images: post condition to be checked.
−
#log error and warning (only the last xxx lines of the console ?) '''or at your convenience the full log of errors. It is open'''
+
#log error and warning (only the last xxx lines of the console ?)
 +
#notify results (failure/fixed) to the lest commiters - Hudson and Jenkins feature
 
#publish results on internet (pass or fail)
 
#publish results on internet (pass or fail)
 +
  
 
===On target tests===
 
===On target tests===
Line 41: Line 42:
 
* Armadeus packages need their own stategy (TBD)
 
* Armadeus packages need their own stategy (TBD)
  
−
==Compilation Test Plan==
+
====Principle:====
−
 
+
−
==On target Test Plan==
+
−
 
+
−
===Principle:===
+
 
* reset the board
 
* reset the board
 
* take control of the target within uboot
 
* take control of the target within uboot
Line 58: Line 55:
 
* publish results on internet
 
* publish results on internet
  
−
===Busybox testsuite - howto===
+
====Busybox testsuite - howto====
  
 
Busibox provides its own testsuite. The following description prepare and run busybox tests:
 
Busibox provides its own testsuite. The following description prepare and run busybox tests:
Line 79: Line 76:
 
...<br>
 
...<br>
  
−
===uclibc test - howto===
+
====uclibc test - howto====
  
 
Seems to require compilation on target. :-(
 
Seems to require compilation on target. :-(
 
* Within 'make menuconfig' -> toolchain enable 'Compile and install uClibc tests'
 
* Within 'make menuconfig' -> toolchain enable 'Compile and install uClibc tests'
  
 +
==Test Plan==
 +
 +
===Compilation Test Plan===
 +
 +
{| border="1" cellpadding="5" cellspacing="0" summary="Compilation tests"
 +
|- style="background:#efefef;"
 +
| '''Test''' || '''description'''  || '''Trigger''' || '''Targets''' || '''Severity'''
 +
|----------------
 +
| '''Source check''' || check all packages for valid download URLs || periodic / daily || main boards || medium / quite easy to find a backup server
 +
|----------------
 +
| '''Source checkout''' ||  Downloads all sources needed for offline-build || periodic / weekly || main boards || medium / quite easy to find a backup server 
 +
|----------------
 +
| '''Build update''' ||  rebuild (incremental) a board on a git change / Check makefile and BR patches || GIt commit  || one board / apf51 || critic / BSP is broken
 +
|----------------
 +
| '''Build boards ''' ||  Offline build boards' main configs  || periodic / daily || main boards || critic / BSP is broken for some boards 
 +
|----------------
 +
| '''Build test config''' ||  Offline build all armadeux test configs || periodic / weekly || test configs || medium
 +
|----------------
 +
| '''Release Source check''' || check all packages for valid download URLs for the current release || periodic / daily || main boards || medium / quite easy to find a backup server
 +
|----------------
 +
| '''Release build boards''' || build boards' main configs with the current release  || periodic / monthly || main boards || critic / BSP release is broken
 +
|----------------
 +
|}
 +
 +
===On target Test===
  
−
==Test cases==
+
postpone..

Latest revision as of 19:51, 1 October 2013

Armadeus Integration Test

Create automatic test & Validation tools

Changelog

september 2013: Installation of a test server with Hudson / Jenkins

Introduction

Test Strategy

  • Production tests: done (out of the scope of this document)
  • Compilation tests

For the time being, compilation tests are manual. The idea is to compile Armadeus project from the head of SVN for each target to check compilation success. success condition is compilation pass and binary images (u-boot, linux, and rootfs) are available in binaries directory. Test could be run every day.

  • On target test run

Here again tests have to be automatized. The idea is to control a target over serial port and ethernet to run test cases and log test results. First priroty is to validate linux kernel, modules and rootfs packages present in the default configuration for the target. Yes, it is doable!


Compilation Tests

Use a build farm to run the tests periodicaly (for each commit). Hudson / jenkins or buildbot are good pretenders. There are many competitors here CI Matrix

  1. checkout the head of GIT
  2. run build for each supported target
  3. check build result
  4. check presence of binary images: post condition to be checked.
  5. log error and warning (only the last xxx lines of the console ?)
  6. notify results (failure/fixed) to the lest commiters - Hudson and Jenkins feature
  7. publish results on internet (pass or fail)


On target tests

Some test tools have to be added to the target to run the tests:

  • capabilities to reset the board (watchdog or wired signal)
  • Add some testsuites to the target rootfs. Some packages like busybox, uclibc provide their own testsuite
  • Armadeus packages need their own stategy (TBD)

Principle:

  • reset the board
  • take control of the target within uboot
  • upload new binary images (with test tools)
  • reset the board
  • pass uboot test. dont know if uboot any test strategy
  • boot linux
  • connect as user root (or test)
  • run integrated testsuites (uclibc, busybox...) and log report
  • run test cases for remaining packages present in the target.
  • Log error and warning
  • publish results on internet

Busybox testsuite - howto

Busibox provides its own testsuite. The following description prepare and run busybox tests:

  • copy busybox .config file to the target rootfs user directory (root or /home/test tbd)
  • copy busybox testsuite directory to the target rootfs (/root or..)
  • build and upload new rootfs to the target (script to be finalized. uboot1.3.4 pathch 3.x provides script "update-all")
  • boot linux and connect as root or user test
  • cd testsuite
  • run shell script: ./testsuite *
  • log test results

screenshoot: > ./runtest *
PASS: basename-does-not-remove-identical-extension
PASS: basename-works
FAIL: bunzip2-reads-from-standard-input
FAIL: bunzip2-removes-compressed-file
FAIL: bzcat-does-not-remove-compressed-file
SKIPPED: bunzip2 (not built)
...

uclibc test - howto

Seems to require compilation on target. :-(

  • Within 'make menuconfig' -> toolchain enable 'Compile and install uClibc tests'

Test Plan

Compilation Test Plan

Test description Trigger Targets Severity
Source check check all packages for valid download URLs periodic / daily main boards medium / quite easy to find a backup server
Source checkout Downloads all sources needed for offline-build periodic / weekly main boards medium / quite easy to find a backup server
Build update rebuild (incremental) a board on a git change / Check makefile and BR patches GIt commit one board / apf51 critic / BSP is broken
Build boards Offline build boards' main configs periodic / daily main boards critic / BSP is broken for some boards
Build test config Offline build all armadeux test configs periodic / weekly test configs medium
Release Source check check all packages for valid download URLs for the current release periodic / daily main boards medium / quite easy to find a backup server
Release build boards build boards' main configs with the current release periodic / monthly main boards critic / BSP release is broken

On target Test

postpone..

Retrieved from "http://armadeus.org/wiki/index.php?title=Armadeus_Integration_Test&oldid=12120"