Meeting 2024-02-28

Attendees: @Michael Mustillo @robert karban @Esma Karagöz @Olivier Cornes

Activity results:

• AI4-17: @Olivier Cornes 's diagram was accepted by me showing the top-level of what he intends to attack with AI. @Esma Karagöz is thinking more generalizing for any analysis of a system, not necessarily aerospace

• The general approach is great, but can we use Olivier’s intent to provide a practical example?

  • Paper reqts need foundation and practical demonstration

  • What aspects should we attack in what order?

Open

Open

• What are the objectives for our paper:

  • Create a SysML v2 model?

  • Create KGs form system descriptions? <--- this is the main part, which needs examples.

  • Should we work backwards from Research Area 1, to the middle blue part?

  • Agree on the achieving the blue part.

  • Don’t forget paper objectives, see Robert message

  • We are aiming for a technical, not vision paper, see again Robert’s message, requires rigorous formal proof:

    • @Esma Karagöz No one in aero industry use an LLM for analysis, only NLP, which provides some novelty

  • Proposed objective:

    • Create several, rigorously proven extractions of correct KGs from system descriptions using LLMs, e.g. aircraft descriptions, etc., that performs in a useful manner for practical purposes, e.g. in safety engineering?

      • We will need to refine this, and then once agreed, move backwards from there to what we need to do.

      • How will we measure this? Precision/recall/F1, etc.

• Use of Gollie restricts us to NER only, per @Olivier Cornes is this the only way?

• What is a profile in our discussions? See also Robert’s SysML diagrams.

• What are the relations we are trying to extract?

• We should focus on creating KGs with next step, after the first paper into real world applications

  • We could use aircraft descriptions to add some practicality

  • How do we deal with adjacency vs zoning in how we are approaching

    • Or is the adjacency exercise simply identification of failure modes, which is also still valuable

    • Maybe we can just find interactions between aircraft functions represented by their components

    • Need to determine how to validate all this

    • @Michael Mustillo @Olivier Cornes to create sample data with ground truths for evaluation

      • We’ll need to agree on how the ground truth looks so we are consistent with each other

      • where to store it? Confluence is terrible

  • @Olivier Cornes will be the Mistral evaluator

  • @Esma Karagöz will be the Gollie evaluator

  • @Michael Mustillo will be the Orca2 evaluator