...
What are the objectives for our paper:
Create a SysML v2 model?
Create KGs form system descriptions? <--- this is the main part, which needs examples.
Should we work backwards from Research Area 1, to the middle blue part?
Agree on the achieving the blue part.
Don’t forget paper objectives, see Robert message
We are aiming for a technical, not vision paper, see again Robert’s message, requires rigorous formal proof:
Esma Karagöz No one in aero industry use an LLM for analysis, only NLP, which provides some novelty
Proposed objective:
Create several, rigorously proven extractions of correct KGs from system descriptions using LLMs, e.g. aircraft descriptions, etc., that performs better than human performance (or at least correct)in a useful manner for practical purposes, e.g. in safety engineering?
We will need to refine this, and then once agreed, move backwards from there to what we need to do.
How will we measure this? Precision/recall/F1, etc.
Use of Gollie restricts us to NER only, per Olivier Cornes is this the only way?
What is a profile in our discussions? See also Robert’s SysML diagrams.
...
What are the relations we are trying to extract?
We should focus on creating KGs with next step, after the first paper into real world applications
We could use aircraft descriptions to add some practicality
How do we deal with adjacency vs zoning in how we are approaching
Or is the adjacency exercise simply identification of failure modes, which is also still valuable
Maybe we can just find interactions between aircraft functions represented by their components
Need to determine how to validate all this
Michael Mustillo Olivier Cornes to create sample data with ground truths for evaluation
We’ll need to agree on how the ground truth looks so we are consistent with each other
where to store it? Confluence is terrible
Olivier Cornes will be the Mistral evaluator
Esma Karagöz will be the Gollie evaluator
Michael Mustillo will be the Orca2 evaluator