View previous topic :: View next topic |
Author |
Message |
Netzman
Joined: 25 Nov 2005 Posts: 127 Location: Graz
|
Posted: Thu May 28, 2015 7:42 pm Post subject: Software unit testing framework / Live debugger |
|
|
Now that I am slowly getting to somewhere, I want to show you what I have so far, I am interested in your opinion and suggestions!
I was tired testing the individual functions of a bigger project again and again by hand when I changed some things in dependant functions, so I began writing a software unit testing framework for Bascom (I got inspired by NUnit, which does a similar thing for .net applications).
The idea is, as mentioned before, to automate the process of testing the code for correct output/behaviour, by comparing the generated/computed data to expected data.
To do this, BUnit (got creative) analyzes, compiles, programs and/or simulates a list of Bascom test codes and communicates with UUT (unit under test) and logs the results (tests passed/failed, test logs etc)
In the Bascom application, a library provides the functions to compare the outputs and to communicate with BUnit over a serial connection.
A test could look like this: Code: | $regfile = "m32def.dat"
$crystal = 16000000
$hwstack = 64
$swstack = 64
$framesize = 64
$baud = 9600
Dim Teststring1 As String * 5 , Teststring2 As String * 5
Dim Testptr As Word , Expectedptr As Word
Dim Testresult As Byte
Dim Testinteger As Integer
$include "inc\BUnit.inc"
Enable Interrupts
Bunit_start
Testinteger = -350
Teststring1 = "test1"
Teststring2 = "test1"
Bunit_message "it's a test!"
Expectedptr = Varptr(teststring1)
Testptr = Varptr(teststring2)
Testresult = Bunit_assert(testptr , Expectedptr , 5)
If Testresult = True Then
'Teststring2 = "test2"
Bunit_breakpoint
Testresult = Bunit_assert(testptr , Expectedptr , 5)
End If
Bunit_end |
The main window after the test run:
The test code can be executed using the Avrora simulator (only Atmega32/64/128), the Bascom internal simulator (or any other simulator that provides a serial interface to the simulated cpu) or in hardware (to program the device, AvrDude is used).
The other part of the application is the live debugger. You can set breakpoints in your program (by calling a sub from the library), on which the program execution is halted and you can view/modify global variables, IO registers and relocate the program counter to continue the execution at another point.
Variable view:
Code view:
It is also possible to completely customize the tests or automate things with scripts. These scripts can be written in VB.net or C#, BUnit compiles the script before executing the rest of the test, in the script you have access to all test-related functions, data and events.
Script example: Code: | Imports System
Imports System.Windows.Forms
Imports Microsoft.VisualBasic
Imports ScriptingInterfaces
Namespace BUnitScripting
Public Class ScriptSample
Implements IScript
Public WithEvents BUnitTest As ScriptingInterfaces.ITest
Public ReadOnly Property Description As String Implements IScript.Description
Get
Return "This is a sample script." & vbCrLf &
"It does nothing."
End Get
End Property
Public Sub Initialize(Test As ITest) Implements IScript.Initialize
BUnitTest = Test
End Sub
Public Sub Dispose() Implements IScript.Dispose
End Sub
Private Sub HandleBeforeTestExecution() Handles BUnitTest.BeforeTestExecution
'MessageBox.Show("Test started!")
End Sub
End Class
End Namespace |
Currently I am working on XMega support, chase bugs here and there and also there are still a few small features left I want to add.
Best regards _________________ LCD Menu | Proportional Fonts |
|
Back to top |
|
|
Meister
Joined: 27 May 2010 Posts: 319
|
Posted: Thu May 28, 2015 10:02 pm Post subject: |
|
|
What is BUnit?
I don't understand.
Regards, Meister |
|
Back to top |
|
|
Netzman
Joined: 25 Nov 2005 Posts: 127 Location: Graz
|
Posted: Fri May 29, 2015 1:10 am Post subject: |
|
|
I try to answer your question.
BUnit is the software unit testing framework/live debugger I described above/I am working on.
Unit testing is a process to ensure your software behaves exactly the way it should. The Wikipedia article explains it in more detail: http://en.wikipedia.org/wiki/Unit_testing
Suppose you wrote a function which does some calculations.
Code: | Function MyFunction (A As Integer, B As Integer) As Integer
A = A + B
MyFunction = A
End Function |
During you develop your application, you change things here and there, on which the calculating function relies.
To test if your function still works correctly, you can write a small test code:
Code: | Dim Result As Integer, Expected As Integer
Result = MyFunction(1, 2)
Expected = 3
If Result = Expected Then
Print "Ok!"
Else
Print "Error!"
End If
|
When you run this test program, you'll see if your function still behaves like intended.
BUnit runs a list of tests automatically and provides you with the results.
It is also possible to communicate with the executed test, together with the scripting feature you could automate also complicated testing procedures.
Or run tests which take hours to complete...
Of course you would not test a function which adds two numbers, but if you have complicated functions which depend on other functions and you have a bunch of them, this could become handy.
And when you run a test and want to know why it fails or want to alter some variables to see if it still fails, that's what the debugger does.
Best regards _________________ LCD Menu | Proportional Fonts |
|
Back to top |
|
|
Meister
Joined: 27 May 2010 Posts: 319
|
Posted: Fri May 29, 2015 9:49 am Post subject: |
|
|
Thanks, Mr. Netzman,
interesting.
So BUnit is your own baby (B for Bascom?), but apparently not yet available. Will you post it here?
Occasionally we find posts here worrying that a (complex) program crashes after running for a longer time, and a reason cannot be pinpointed. So Unit Testing might help.
From Wikipeadia: "Unit tests are short code fragments"
But many problems just arise from the interaction of code fragments....
and most serious problems (to my view) arise from using interrupts...
Couldl you handle this?
Of course, each code segment should be tested that way as a prerequisite for the functioning of the whole.
Regards, Meister |
|
Back to top |
|
|
Netzman
Joined: 25 Nov 2005 Posts: 127 Location: Graz
|
Posted: Fri May 29, 2015 2:45 pm Post subject: |
|
|
Meister wrote: | So BUnit is your own baby (B for Bascom?), but apparently not yet available. Will you post it here? |
Yes, of course! As I wrote in the first post, I am still working on some small features and bug fixes, but most of the things I wanted to do are working.
Meister wrote: | Occasionally we find posts here worrying that a (complex) program crashes after running for a longer time, and a reason cannot be pinpointed. So Unit Testing might help.
From Wikipeadia: "Unit tests are short code fragments"
But many problems just arise from the interaction of code fragments....
and most serious problems (to my view) arise from using interrupts...
Couldl you handle this? |
Short: No, but sometimes yes.
More detailed: Unit testing targets the behaviourial aspects of the software, not the syntactical or technical. So your software must be able to execute - since the test related/debugging functions are also implemented in software, executed by the DUT (device under test). Most of the functionality of BUnit is implemented in the serial receive ISR, so it relies on the correct execution of interrupts.
On the other hand, you can execute the test (that fails occassionally) using the Bascom simulator, and try to track the problem. When the test is executed on the hardware, it is hard to track variable values etc, with the debugger you could directly read or alter them...
It really depends on the nature of the problem, you can look at it as an additional tool to help improving the software quality, not a complete debugging solution.
Meister wrote: | Of course, each code segment should be tested that way as a prerequisite for the functioning of the whole. |
That's a good point, if you can make sure, your base functions do exactly what you want, in every possible case (to connect to the example from before, you would not simply test MyFunction to be able to add 1 and 2 correctly, you would "sweep" the whole range of possible values), then you can avoid lots of errors you may encounter in the high level functions.
Additionally, when you write your functions and in paralell you develop a test scenario for this function, you can avoid some errors before even testing it.
The Assert-function (which does the comparison result/expected) itself is kept simple, it just takes two memory addresses and the data length, if they are identical, the assertion is passed, else it fails.
But on top of that, you can implement more complicated sanity checks, by writing your own pass/fail logic and set a variable to 0/1, which then is checked by the Assert-function.
A more sophisticated and specific example of what you could do, you built a motor controller board (or temperature control or whatever) with a PID-control loop and you have a complete test-setup on your desk (board, motor, serial connection to PC). You start the BUnit test run, it compiles and programs the device and performs the test.
Your test code first measures the response of the control loop (sweep the output, measure the feedback) and sends the gathered data to the PC, where you execute the BUnit test with a script. The script takes the measured data, calculates the control loop parameters and sends them back to the motor control board (Or you skip the automatic control loop tuning and enter the parameters by hand).
Then your test procedure begins, where you set the value for your PID control and check if the loop stays in a defined window of normal operation. If your loop leaves your safety window, the test fails.
You could also run other things in paralell to the control loop test on your board, to check if there are some unwanted interactions in your code parts.
Basically, you can test anything that you can check for sanity in your code (not limited to software-only code, you could also test hardware-related functions).
Of course, if you have only one feature to test it might be overkill to implement an automated test procedure, in this case it might be more practical to just test it the "traditional way".
But in a bigger project, you could easily make 10 or 20 test cases, which then makes your life easier to test all these things (especially during ongoing development, when you need to test things again when you changed something).
Best regards _________________ LCD Menu | Proportional Fonts |
|
Back to top |
|
|
Meister
Joined: 27 May 2010 Posts: 319
|
Posted: Fri May 29, 2015 10:31 pm Post subject: |
|
|
Looking forward to see Bunit...
I just (and quite often) had trouble with variable overflow which easily crops up when using Tiny Avr's because of the need to keep variables short..
Regards, Meister |
|
Back to top |
|
|
albertsm
Joined: 09 Apr 2004 Posts: 5913 Location: Holland
|
Posted: Sat May 30, 2015 9:50 pm Post subject: |
|
|
Hi Netzman
it looks like a great tool. unit testing is very important when you need to maintain a lot of code.
you have explained it well. good to see your projects. _________________ Mark |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum You cannot attach files in this forum You cannot download files in this forum
|
|