1# Parameter Framework (PF) test plan
2
3## Notes
4- The unit test code MAY be able to list implemented scenarios
5  in this same BDD format in order to check scenario coverage.
6- All scenarios implicitly start by _"GIVEN a parameter framework"_.
7- If a scenario does not specify a config files,
8  the following minimal files SHOULD be used:
9
10Top level file (replace `{structurePath}` by its value):
11~~~ xml
12<?xml version='1.0' encoding='UTF-8'?>
13<ParameterFrameworkConfiguration SystemClassName='test'
14                                 TuningAllowed='false' ServerPort='1'>
15    <SubsystemPlugins/>
16    <StructureDescriptionFileLocation Path='{structurePath}'/>
17</ParameterFrameworkConfiguration>
18~~~
19
20Structure file:
21~~~ xml
22<?xml version='1.0' encoding='UTF-8'?>
23<SystemClass Name='test'>
24    <Subsystem Name='test' Type='Virtual'>
25        <ComponentLibrary/>
26        <InstanceDefinition>
27            <BooleanParameter Name="placeholder"/>
28        </InstanceDefinition>
29    </Subsystem>
30</SystemClass>
31~~~
32
33## Starting the Parameter Framework
34
35- set a logger
36    - [X] Scenario: Default logger
37        * WHEN no logger is set
38        * THEN start should succeed
39
40    - [X] Scenario: No logger
41        * WHEN a nullptr logger is set
42        * THEN start should succeed
43
44    - [X] Scenario: Logger should receive info and warnings
45        * GIVEN config files that emit warnings
46        * GIVEN a logger that store logs
47        * WHEN the record logger is set
48        * THEN start should succeed
49        * AND_THEN the logger should have stored info and warning log
50
51    - [X] Scenario: Unset logger
52        * GIVEN config files that emit warnings
53        * GIVEN a logger that store logs
54        * WHEN the record logger is set
55        * AND_WHEN a nullptr logger is set
56        * THEN start should succeed
57        * AND_THEN the record logger should NOT have stored any info or warning log
58
59- set/get various properties before startup:
60    - "force no remote interface"
61        - [ ] Scenario: Tuning OK
62            * GIVEN config files with tuning enabled on a valid port
63            * WHEN the remote interface is NOT forbiden
64            * THEN start should succeed
65
66        - [ ] Scenario: Invalid tunning
67            * GIVEN config files with tuning enabled on an invalid port (0?)
68            * WHEN the remote interface is NOT forbiden
69            * THEN start should fail
70
71        - [ ] Scenario: Forbiden invalid tunning
72            * GIVEN config files with tuning enabled on an invalid port (0?)
73            * WHEN the remote interface is forbiden
74            * THEN start should succeed
75
76    - "failure on missing subsystem"
77        - [ ] Scenario: Plugin OK
78            * GIVEN config files with a valid plugin
79            * WHEN the missing subsystem policy is set to failure
80            * THEN start should succeed
81
82        - [X] Scenario: Wrong plugin
83            * GIVEN config files with a invalid plugin
84            * WHEN the missing subsystem policy is set to failure
85            * THEN start should fail
86
87        - [ ] Scenario: Wrong plugin but fallback
88            * GIVEN config files with a invalid plugin
89            * WHEN the missing subsystem policy is set to success
90            * THEN start should succeed
91
92    - "failure on failed settings load"
93        - [X] Scenario: Settings OK
94            * GIVEN config files with a valid settings file
95            * WHEN settings load failure is requested to abort start
96            * THEN start should succeed
97
98        - [X] Scenario: Wrong settings
99            * GIVEN config files with a invalid (non existant?) settings file
100            * WHEN settings load failure is requested to abort start
101            * THEN start should fail
102
103        - [X] Scenario: Wrong settings but ignore
104            * GIVEN config files with a invalid (non existant?) settings file
105            * WHEN settings load failure is requested to abort start
106            * THEN start should succeed
107
108    - "schema folder location" and "validate schema on start"
109        - [ ] Scenario: Schema OK
110            * GIVEN config files with correct default schema location
111            * WHEN schema folder location is left to default
112            * WHEN schema validation is enabled
113            * THEN start should succeed
114
115        - [ ] Scenario: Inexisting schemas
116            * GIVEN config files with correct default schema location
117            * WHEN schema folder location is set to an invalid location (/doesNotExist ?)
118            * WHEN schema validation is enabled
119            * THEN start should fail
120
121        - [ ] Scenario: Inexisting ignored schemas
122            * GIVEN config files with correct default schema location
123            * WHEN schema folder location is set to an invalid location (/doesNotExist ?)
124            * WHEN schema validation is disabled
125            * THEN start should succeed
126
127        - [ ] Scenario: Inexisting used schemas
128            * GIVEN config files with custom schema location (./shemaCustomFolderName)
129            * WHEN schema folder location is set to the custom shema location
130            * WHEN schema validation is enabled
131            * THEN start should succeed
132
133        - [ ] Scenario: Schema OK, config KO
134            * GIVEN incorect config files with correct default schema location
135            * WHEN schema validation is enabled
136            * THEN start should fail
137
138### Error cases/special cases
139
140- invalid configuration or a file can't be read:
141    - [X] Scenario: invalid top level config file
142        * GIVEN an incorect (empty ? wrong balisa ?) config top level file.
143        * THEN start should fail
144
145    - [X] Scenario: invalid structure
146        * GIVEN an incorect (empty ? wrong balisa ?) structure file.
147        * THEN start should fail
148
149    - [X] Scenario: invalid settings
150        * GIVEN an incorect (empty ? wrong balisa ?) settings file.
151        * THEN start should fail
152
153- plugins can't be found (unless silenced by the user)
154    - Done in "failure on missing subsystem"
155
156- usage of `<xi:include .../>`
157    - [ ] Scenario: Arbitrary xincluded node
158        * GIVEN a top level file xincluding a file with the "SettingsConfiguration" balisa
159        * THEN start should succeed
160
161- plugins:
162    - non-empty "Location" attribute
163        - [ ] Scenario: Plugin location OK
164            * GIVEN a top level file with a plugin folder location attribute
165            * THEN start should succeed
166    - empty "Location" attribute
167        - relative path
168            - [ ] Scenario: Empty Location, standard plugin install
169                * GIVEN config files with an empty location attribute
170                * GIVEN a plugin installed in a dlopen standard path
171                * GIVEN that "Location/Plugin[name]" is the plugin name
172                * THEN start should succeed
173
174            - [ ] Scenario: Empty Location, non standard plugin install
175                * GIVEN config files with an empty location attribute
176                * GIVEN a plugin NOT installed in a dlopen standard path
177                * GIVEN that "Location/Plugin[name]" is the plugin name
178                * THEN start should fail
179
180        - absolute path
181            - [ ] Scenario: Empty Location, plugin absolute path
182                * GIVEN config files with an empty location attribute
183                * GIVEN a plugin
184                * GIVEN that "Location/Plugin[name]" is the plugin absolute path
185                * THEN start should succeed
186
187            - [ ] Scenario: Empty Location, plugin non existing path
188                * GIVEN config files with an empty location attribute
189                * GIVEN that "Location/Plugin[name]" is an non existing path (/libDoesNotExist.so?)
190                * THEN start should fail
191
192* * *
193
194## Logging
195
196- info and warning
197    - Partly covered by the "set a logger" section
198    - [ ] Scenario: Plugin info and warning log
199        * GIVEN a plugin that log a unique string as info and warning
200        * GIVEN a logger that store logs
201        * WHEN the record logger is set
202        * THEN start should succeed
203        * AND_THEN the logger should have stored the plugin specific info and warning log
204
205* * *
206
207## Basic Domains & Configurations management
208
209- [ ] Scenario: create and delete an empty domain
210    * GIVEN a started parameter framework without domains
211    * THEN creating a domain with an arbitrary name ("domain1"?) should succeed
212    * (AND_THEN the domain list should contain the domain name)
213    * THEN deleting the domain should succeed
214
215- [ ] Scenario: create and delete a configuration
216    * GIVEN a started parameter framework with a domain
217    * THEN creating a configuration in this domain with an arbitrary name ("config1") should succeed
218    * (AND_THEN the domain configuration list should contain the configuration name)
219    * THEN deleting the configuration should succeed
220    * (AND_THEN the domain configuration list should not contain the configuration name)
221
222- [ ] Scenario: delete a domain with configurations
223    * GIVEN a started parameter framework with a domain and several configuration (3?)
224    * THEN deleting the domain should succeed
225    * (AND_THEN the domain list should not contain the domain name)
226
227- [ ] Scenario: delete all domains
228    * GIVEN a started parameter framework with several domains
229    * THEN deleting all domains should succeed
230    * (AND_THEN the domain list should be empty)
231
232- [ ] Scenario: rename a domain
233    * GIVEN a started parameter framework with a domain and several configuration (3?)
234    * THEN renaming the domain to an arbitrary name should succeed
235
236
237### Error cases/special cases
238
239- [ ] Scenario: create a domain with an already-existing name
240    * GIVEN a started parameter framework with a domain (arbitrary name)
241    * THEN creating a second domain with the same name than the first should fail
242
243- [ ] Scenario: create a configuration with an already-existing name
244    * GIVEN a started parameter framework with a domain
245            containing one configuration (arbitrary name)
246    * THEN creating a second configuration in this domain
247           with the same name than the fist should fail
248
249- [ ] delete a non-existent domain
250    * GIVEN a started parameter framework without domain
251    * THEN deleting an arbitrary named domain should fail
252
253- [ ] delete a non-existent configuration
254    * GIVEN a started parameter framework one domain
255    * THEN deleting in this domain a arbitrary named configuration should fail
256
257- [ ] list configuration of a unknown domain
258    * GIVEN a started parameter framework without domain
259    * THEN listing configuration of an arbitrary named domain should fail
260
261- [ ] Scenario: rename a domain to an already used name
262    * GIVEN a started parameter framework with two domains
263            with arbitrary different names ("domain1" and "domain2")
264    * THEN renaming first domain ("domain1") to the name of the second one ("domain2") should fail
265
266- [ ] Scenario: rename a configuration to an already used name
267    * GIVEN a started parameter framework with one domain containing two configuration
268            with arbitrary different names ("config1" and "config2")
269    * THEN renaming first configuration ("config1")
270           to the name of the second one ("config2") should fail
271
272- [ ] Scenario: rename a domain to its own name
273    * GIVEN a started parameter framework one arbitrary named domain
274    * THEN renaming the domain to its own name should succeed
275
276- [ ] rename a configuration to its own name
277    * GIVEN a started parameter framework a domain and one configuration
278    * THEN renaming the configuration to its own name should succeed
279
280* * *
281
282## Domain modification
283
284- set/get sequence awareness
285- add/list/remove elements to a domain
286- *(split domain ?)*
287- *(listBelongingDomains ?)*
288- *(listAssociatedDomains ?)*
289- *(listAssociatedElements ?)*
290- *(listConflictingElements ?)*
291- *(listRogueElements ?)*
292
293### Error cases/special cases
294
295- add (or remove) a non-existent element
296- add (or remove) a parameter to (or from) a non-existent domain
297
298* * *
299
300## Configurations modification
301
302- set a configuration rule; get it back
303    - trivial rules, complex rules (fuzzing ?)
304    - clear a rule
305- save/restore a configuration
306- set/get element sequence
307- set/get a parameter for a given configuration
308- set a parameter belonging to a domain and save a configuration
309
310### Error cases
311
312- set a wrongly-formatted rule
313- set a rule using an unknown criterion (or unknown criterion value)
314- set/get a rule to a non-existent domain (or configuration)
315
316* * *
317
318## Criteria
319
320- create a criterion
321    - exclusive
322    - inclusive
323- *(list criteria)*
324- set a criterion; get a criterion value
325    - exclusive criterion
326    - inclusive criterion
327        - special case: "none" value
328
329### Error cases/special cases
330
331- duplicated criterion name (with the same type/with a different type)
332- duplicated criterion value
333- max supported number of criterion values
334
335* * *
336
337## Settings Import/Export
338
339- import xml domains
340    - w/ or w/o settings
341    - from a file/from a string
342    - all/single
343- export xml domains
344    - w/ or w/o settings
345    - to a file/from a string
346    - all/single
347
348### Error case/special cases
349
350- invalid settings
351    - invalid xml (should we use a fuzzing framework ?)
352    - valid xml but not matching the XSD (ditto)
353    - valid but unusable data:
354        - unknown criterion/criterion values
355        - unknown parameters
356- unreadable/unwritable file
357- single domain import overwrite
358    - fail if overwrite not requested
359    - succeed if overwrite requested
360
361* * *
362
363## Parameters
364
365- structures are correctly instantiated
366    - parameters in the "instance definition"
367    - component types instantiation
368    - parameter blocks
369    - bit blocks
370    - arrays
371- check parameter mappings are correctly applied
372    - at all levels possible (subsystem, component type, component instance,
373            block, parameter...)
374    - context mappings, amends, instantiation mappings
375- set/get a parameter
376- parameter handles
377- test on parameter values
378    - booleans
379    - bit blocks
380    - numeric types ranges
381    - fixed-point tests
382    - string length
383
384* * *
385
386## Configuration application
387
388- apply configurations
389    - sequence aware domains
390- forced sync
391
392* * *
393
394## Misc/to be classified
395
396- tuning mode
397- auto sync
398- remote interface instantiation
399- value space (raw/real)/format (dec/hex)
400
401### Error cases
402
403