I have a command line app (called by Firefox) that reads Excel files created by the Fidelity stock screener (logged in via Firefox) and writes them to an XML WL6 DataSet. It also reads existing WL6 DataSets and removes incoming screener duplicates so existing WL6 stocks aren't screened twice. It uses the XMLserializer and deserializer to read and write all WL6 XML DataSets.
I now want to convert it to WL8. But in reading the DataSetProvider and DataSet class docs, I'm unable to find methods that actually do the reading and writing of WL8 DataSet symbols. Where are they?
Also, are all the methods for the DataSetProvider and DataSet classes available to the command line app even if WL8 isn't running? I'm assuming they are; otherwise, what good are they?
Finally, I asking myself if I should simply use StreamWriter and StreamReader along with the XMLserializer and deserializer to manage these screener DataSets as I have done for years with WL6? Would that be simplest? If so, can you point to the schema file the New DataSet Wizard employs for the XMLserializer/deserializer. I want to employ the simplest XML schema possible because these screener DataSets will only contain symbols--nothing more.
I now want to convert it to WL8. But in reading the DataSetProvider and DataSet class docs, I'm unable to find methods that actually do the reading and writing of WL8 DataSet symbols. Where are they?
Also, are all the methods for the DataSetProvider and DataSet classes available to the command line app even if WL8 isn't running? I'm assuming they are; otherwise, what good are they?
Finally, I asking myself if I should simply use StreamWriter and StreamReader along with the XMLserializer and deserializer to manage these screener DataSets as I have done for years with WL6? Would that be simplest? If so, can you point to the schema file the New DataSet Wizard employs for the XMLserializer/deserializer. I want to employ the simplest XML schema possible because these screener DataSets will only contain symbols--nothing more.
Rename
Check out this code sample and Post #1:
https://www.wealth-lab.com/Discussion/Update-DataSets-property-in-DataSetProvider-while-WL-is-running-8173
https://www.wealth-lab.com/Discussion/Update-DataSets-property-in-DataSetProvider-while-WL-is-running-8173
So after reading the code in the link above, you're saying base.Initialize() writes this.dataSet to disk and sets up an event so whenever this.dataSet changes, it will automatically update on disk. Is this correct? (That's a nice feature.)
Now how can I read a dataset into my command line app? What is the way to associate a group of existing WL8 symbols to a given dataset name? (I use a regexp on the dataset name to filter existing symbols against incoming screener symbols.)
And does my command line app have access to all these methods by "using WealthLab.Core" even if WL8 isn't running? Remember, the command line app is invoked by Firefox while logged into Fidelity.
Now how can I read a dataset into my command line app? What is the way to associate a group of existing WL8 symbols to a given dataset name? (I use a regexp on the dataset name to filter existing symbols against incoming screener symbols.)
And does my command line app have access to all these methods by "using WealthLab.Core" even if WL8 isn't running? Remember, the command line app is invoked by Firefox while logged into Fidelity.
Don't overcomplicate and do all the processing in the WL8 DataSetProvider.
Are you saying the DataSetProvider should log into Fidelity website and execute the Fidelity screener scripts (I have three screener scripts there.) to produce the Excel files? That's way too complicated for me. I don't know how to do that. Let's have Firefox do that part.
All Firefox needs to do is log onto Fidelity and run the screener scripts to create the Excel files. Then Firefox calls my command line program to process those Excel files. Anything else is over my head. I'm not that skilled at website scraping.
So what about the answers to my questions in Post #2?
All Firefox needs to do is log onto Fidelity and run the screener scripts to create the Excel files. Then Firefox calls my command line program to process those Excel files. Anything else is over my head. I'm not that skilled at website scraping.
So what about the answers to my questions in Post #2?
Just my two cents: either get rid of this legacy code involving Firefox and console stuff or simply stick to your tried and true routine.
It seems to me that the DataSet Provider API is missing info about DataSetFactory (or in general, a reference to FactoryBase methods) that could help you. I haven't really worked much with WL DataSets, but it looks to me like you can save modifications to a DataSet by calling DataSetFactory.Instance.Modify(DataSet ds).
@Glitch - am I off about this?
@Glitch - am I off about this?
Sure, the Stock Scanner DataSetProvider calls Modify since it allows to make changes to its screens on-the-fly while WL8 is running. That of course is feasible.
However, it seems that's topic starter's biggest concern is porting his complex legacy code. And if Firefox can login to Fido to get those Excel files, a scraper could do it too.
However, it seems that's topic starter's biggest concern is porting his complex legacy code. And if Firefox can login to Fido to get those Excel files, a scraper could do it too.
The absolute simplest solution is for you to give me the schema type name the New DataSet Wizard uses to create its XML datasets. Very simple question.
I need to weed out the screener symbols that already exist in WL8 datasets, and I use a regexp against the existing dataset names to pick which datasets for this weeding out process. Here's my existing XMLserializer code for WL6 (which has been running for 7 years).
If those methods are accessible to my command line app, that works too. That's an interesting option.
I would like to avoid writing a complicated website scraper to login into the Fidelity screener and manipulate all the screener scripts to generate the Excel files that Firefox is currently doing. That's outside my expertise (although it might be something to do in the future with your help).
I need to weed out the screener symbols that already exist in WL8 datasets, and I use a regexp against the existing dataset names to pick which datasets for this weeding out process. Here's my existing XMLserializer code for WL6 (which has been running for 7 years).
CODE:
static void ReadXmlDataSet(FileInfo inputFile, ref SortedSet<string> symbols, bool flagDuplicates) { if (inputFile.Exists) { XmlSerializer xs = new XmlSerializer(typeof(DataSet)); StreamReader sr = new StreamReader(inputFile.OpenRead()); DataSet dataSet = (DataSet)xs.Deserialize(sr); sr.Close(); char[] delimiters = new char[] {','}; foreach (string symbol in dataSet.DSString.Split(delimiters, StringSplitOptions.RemoveEmptyEntries)) if (!symbols.Add(symbol) && flagDuplicates) Console.WriteLine(string.Format("Duplicate symbol {0} found in dataset: {1}", symbol, inputFile.Name)); } } static void WriteXmlDataSet(FileInfo outputFile, SortedSet<string> symbols) { DataSet dataSet = new DataSet { Name = Path.GetFileNameWithoutExtension(outputFile.Name), BarInterval = "0", //no intraday divisions Scale = "Daily", //daily bars ProviderName = "IQFeedStaticProvider", DSString = String.Join(",", symbols) //concatenate symbol list }; XmlSerializer xs = new XmlSerializer(typeof(DataSet)); StreamWriter sw = new StreamWriter(outputFile.Create()); //overwrite any existing file xs.Serialize(sw, dataSet); sw.Close(); } [XmlRoot(IsNullable = false)] [XmlType(AnonymousType = true)] public class DataSet { [XmlElement(Form = System.Xml.Schema.XmlSchemaForm.Unqualified)] public string Name { get; set; } [XmlElement(Form = System.Xml.Schema.XmlSchemaForm.Unqualified)] public string Scale { get; set; } [XmlElement(Form = System.Xml.Schema.XmlSchemaForm.Unqualified)] public string BarInterval { get; set; } [XmlElement(Form = System.Xml.Schema.XmlSchemaForm.Unqualified)] public string DSString { get; set; } [XmlElement(Form = System.Xml.Schema.XmlSchemaForm.Unqualified)] public string ProviderName { get; set; } }
QUOTE:
... DataSet Provider API is missing info about DataSetFactory (or in general, a reference to FactoryBase methods) ...
If those methods are accessible to my command line app, that works too. That's an interesting option.
I would like to avoid writing a complicated website scraper to login into the Fidelity screener and manipulate all the screener scripts to generate the Excel files that Firefox is currently doing. That's outside my expertise (although it might be something to do in the future with your help).
Your Response
Post
Edit Post
Login is required