Foundation Model Evaluator 

In this tab, you can find all the relevant information for your submission. The following contents must be submitted for participation

How to submit

1. Submission template (mandatory)
   A standard template is available. The template will help you structure your solutions and approaches and provide answers to all relevant questions and frameworks. Download the Submission Template here
2. Optional Content
   You can try to stand out and also supplement your submission with prototypes, image and video materials, or other concepts or visual elements - simply submit these files additionally via our platform.

Key Guidelines: 

Must-haves for the initial submission:

1. Evaluation Metrics: Participants should identify and implement important metrics for assessing or evaluating AI foundation model performance. While considering efficiency, accuracy, the rate of hallucination and cost, innovators are encouraged to determine relevant metrics that are crucial. 
2. Relevant Foundation Models: Submissions should focus on the most common models available now. On-top open-source models are of particular interest, but participants should consider the full spectrum of available foundation models. 

Optional criteria for the long-term implementation: While these guidelines may not need to be fully met in the initial application, they should be considered for the long-term development: 

1. Use Case Compatibility: Solutions should be capable of evaluating distinct use cases, with requirements defined by users. See the 'Knowledge Base' tab for examples relevant to Stellantis. 
2. Feedback Mechanism: In the long-term solutions should be able to establish a user feedback mechanism that contributes to the continuous refinement of the models. 
3. Automation and Scalability: Initial concepts may initially require manual work & input, with a vision to automate these processes in later stages. 
4. Flexibility: Design the tool to easily incorporate new metrics or data sources as they become available, ensuring it remains relevant and effective. 

Submission Criteria

Participants are asked to submit a detailed concept or an initial prototype that addresses the outlined problem statements. Submissions should consider: 

Participant 

• Personal or Team Profile: Information on yourself, your team or your company’s expertise and experience. 
• Past Success Stories: Examples of previous successful initiatives in the AI space 

Solution 

• Evaluation Metrics: What metrics do you employ to evaluate the effectiveness of foundation models? 
• Model Market: Which current market-available foundation models have you evaluated or considered, and why are they suitable for Stellantis's requirements? 
• Scalability and Adaptability: Explain how your solution can 1) scale and evolve alongside the different Stellantis use cases and 2) adapt to the shifting foundation model and AI landscape. How can user feedback be included in this development? 
• Prototype Development: If applicable, provide a prototype or at least a detailed concept that demonstrates your approach. 
• Transition to Automation: How do you envision transitioning your concept or initial prototype into a fully automated solution within the Stellantis environment? 

Further partnership 

• Projected Timeline and Budget: Suggest an estimate of the timeline required for testing a first prototype within Stellantis. How can we build a functional POC in the 6-month following the initial deadline? 
• Partnership Model Proposal: Suggestions for a potential business or research partnership model with Stellantis, outlining mutual benefits and responsibilities. 

For more information on foundation model use cases click on the tab "Knowledge Base".