/ notes / ai

⇒ This is just a collection of notes, not a structured or comprehensive document. ⇐

Comparison of different AI models performing the same task

---------------------
### DRAFT VERSION ###
---------------------

TL;DR: different AI models implementing a migration plan for porting a simple WinForms application to .NET 10 + WPF

WinForm (original)

original

.NET 10 + WPF (‘GPT 5.3 Codex’ test case)

original

Tests

Elements:

a very simple WinForms app: https://github.com/fhtino/stuff/tree/main/HttpsCertAlert
Visual Studio Code 1.124.2
GitHub Copilot Pro subscription

Target: porting the app to .NET 10 + WPF

Step zero: created a migration plan using VS Code + plan agent + Claude Sonnet 4.6. The plan was saved as an MD file inside the repository.

For each tested model, I created a git branch (named after the AI model used). Then I selected the model on the chat (a new one chat for every test), opened the migration_plan.md file and I entered always the same prompt: implement migration plan.

Results

Model	Input tokens	Cached tokens	Output tokens	Credits	Result
AUTO –> GPT 5.4 mini	?	?	?	24.9	OK
MAI-Code-1-Flash / Medium	24.000	2.000.000	18.200	31.7	APP RUNTIME ERROR
MAI-Code-1-Flash / High	43.100	2.045.000	33.100	34.7	APP RUNTIME ERROR
Claude Haiku 4.5	31.000	1.810.000	22.000	41.5	APP RUNTIME ERROR
Gemini Flash 3.5 / Medium	216.000	1.408.000	13.000	65.6	APP RUNTIME ERROR
GPT 5.4 Medium	50.800	487.000	16.500	49.7	OK
AUTO (2) –> Haiku	27.700	2.140.000	21.300	40.8	OK
GPT 5.3 Codex / Medium	26.200	580.800	12.900	32.8	OK
MAI-Code-1-Flash / Medium (2)	35.000	1.580.000	16.200	22.8	APP RUNTIME ERROR

(?): I forgot to save the token numbers for this test.

Run details

AUTO : OK

Model used: GPT 5.4 mini 24.9 credits

MAI-Code-1-Flash / Medium

Result: runtime error, running the app.

It required a couple of more prompts to remove old files and clean-up solution.

    IN     CACHED    OUT
000  1.500.000  15000  26.1
000    250.000   1400   2.7
800    250.000   1800   2.9

Runtime error:

Unhandled exception. System.NotSupportedException: This type of CollectionView does not support changes to its SourceCollection from a thread different from the Dispatcher thread.
   at System.Windows.Data.CollectionView.OnCollectionChanged(Object sender, NotifyCollectionChangedEventArgs args)
   at System.Collections.ObjectModel.ObservableCollection`1.OnCollectionChanged(NotifyCollectionChangedEventArgs e)
   at HttpsCertAlert.MainWindow.RefreshGrid() in XXXXX\HttpsCertAlert\HttpsCertAlert\MainWindow.xaml.cs:line 94
   at HttpsCertAlert.MainWindow.CheckTimer_Elapsed() in XXXXX\HttpsCertAlert\HttpsCertAlert\MainWindow.xaml.cs:line 55
   at HttpsCertAlert.MainWindow.<Window_Loaded>b__8_0(Object s, ElapsedEventArgs args) in XXXXX\HttpsCertAlert\HttpsCertAlert\MainWindow.xaml.cs:line 37

MAI-Code-1-Flash / High

Result: runtime error, running the app.

39500 / 1750000 / 32500  /  31.9
 3600 /  295000 /   600  /   2.8

Runtime error:

Unhandled exception. System.NotSupportedException: This type of CollectionView does not support changes to its SourceCollection from a thread different from the Dispatcher thread.
   at System.Windows.Data.CollectionView.OnCollectionChanged(Object sender, NotifyCollectionChangedEventArgs args)
   at System.Collections.ObjectModel.ObservableCollection`1.OnCollectionChanged(NotifyCollectionChangedEventArgs e)
   at HttpsCertAlert.MainWindow.RefreshChecks() in XXXXX\HttpsCertAlert\HttpsCertAlert\MainWindow.xaml.cs:line 64
   at HttpsCertAlert.MainWindow.CheckTimer_Elapsed(Object sender, ElapsedEventArgs e) in XXXXX\HttpsCertAlert\HttpsCertAlert\MainWindow.xaml.cs:line 38

Claude Haiku 4.5

Result: runtime error, running the app.

31000 / 1810000 / 22000 / 41.5

Unhandled exception. System.Windows.Markup.XamlParseException: 'Provide value on 'System.Windows.Baml2006.TypeConverterMarkupExtension' threw an exception.' Line number '41' and line position '25'.
 ---> System.IO.IOException: Cannot locate resource 'resources/lock_blue.ico'.
   at MS.Internal.AppModel.ResourcePart.GetStreamCore(FileMode mode, FileAccess access)
   at System.IO.Packaging.PackagePart.GetStream(FileMode mode, FileAccess access)
   at MS.Internal.IO.Packaging.PackagePartExtensions.GetSeekableStream(PackagePart packPart, FileMode mode, FileAccess access)
   at System.IO.Packaging.PackWebResponse.CachedResponse.GetResponseStream()
   at System.Windows.Media.Imaging.BitmapDecoder.SetupDecoderFromUriOrStream(Uri uri, Stream stream, BitmapCacheOption cacheOption, Guid& clsId, Boolean& isOriginalWritable, Stream& uriStream, UnmanagedMemoryStream& unmanagedMemoryStream, SafeFileHandle& safeFilehandle)

Gemini Flash 3.5 / Medium

Result: runtime error, running the app.

216000 / 1408000 / 13000 / 65.6

Unhandled exception. System.InvalidOperationException: A TwoWay or OneWayToSource binding cannot work on the read-only property 'Active' of type 'HttpsCertAlert.CheckViewModel'.
   at MS.Internal.Data.PropertyPathWorker.CheckReadOnly(Object item, Object info)
   at MS.Internal.Data.PropertyPathWorker.ReplaceItem(Int32 k, Object newO, Object parent)
   at MS.Internal.Data.PropertyPathWorker.UpdateSourceValueState(Int32 k, ICollectionView collectionView, Object newValue, Boolean isASubPropertyChange)
   at MS.Internal.Data.ClrBindingWorker.AttachDataItem()
   at System.Windows.Data.BindingExpression.Activate(Object item)

GPT 5.4 Medium

Result: OK

50800 / 487000 / 16500 / 49.7 credit

AUTO (2)

Result OK but UI top elements moved to bottom

Automatically chosen model: Haiku

(executed the app)

27700 / 2140000 / 21300 / 40.8

GPT 5.3 Codex / Medium

Result: OK (very good)

26200 / 580800 / 12900 / 32.8 credits

MAI-Code-1-Flash / Medium (2) (no copilot_instructions.md)

Remove no copilot_instructions.md before executing the prompt.

Result: runtime error, running the app.

35000 / 1580000 / 16200 / 22.8

Unhandled exception. System.NotSupportedException: This type of CollectionView does not support changes to its SourceCollection from a thread different from the Dispatcher thread.
   at System.Windows.Data.CollectionView.OnCollectionChanged(Object sender, NotifyCollectionChangedEventArgs args)
   at System.Collections.ObjectModel.ObservableCollection`1.OnCollectionChanged(NotifyCollectionChangedEventArgs e)
   at HttpsCertAlert.MainWindow.RefreshChecks() in XXXXX\HttpsCertAlert\HttpsCertAlert\MainWindow.xaml.cs:line 59
   at HttpsCertAlert.MainWindow.CheckTimer_Elapsed() in XXXXX\HttpsCertAlert\HttpsCertAlert\MainWindow.xaml.cs:line 48
   at HttpsCertAlert.MainWindow.<.ctor>b__5_0(Object _, ElapsedEventArgs _) in XXXXX\HttpsCertAlert\HttpsCertAlert\MainWindow.xaml.cs:line 30