Roadmap and an architecture to fulfill it

[Joost van Baal]

On Fri, Jun 15, 2001 at 01:31:32PM -0700, Christoph Lameter wrote:
> I have talked with Francis a lot today on IRC on how to reach the goals.
> A key element is the ease of handling and the configuration of the whole
> system. Many of the components are the typical import,export,generate
> summaries/report stuff known from databases. That why we thought it would
> be useful to store the data in a SQL database 

An SQL database allows a relational database.  Currently, we use only
_one_ table.  If we go for an sql database, we have to consider this.
Is it usefull to use sql, even though we're not using multiple tables?
Or would we like to _go_ using multiple tables?

> and use queries to retrieve
> data. Francis would generate/find/customize and XML interpreter that
> allows the embedding of SQL statements in the XML description of the
> report.
> 
> All reports should be defined by one script with the following parameters:
> 
> lr_report <output format> <type-of-report> <reportparameters>
> 
> This script would be invokable from the commandline or from a webpage.

The issue here is of course what to allow in <reportparameters>.

> Using a SQL database would mean that we already have a powerful query
> language doing aggregation, selecting, sorting and all the other stuff
> needed for effective data processing. Each of the DLF service structures
> would require a corresponding SQL table to be generated. Databases are
> generated by customer. 

What do you mean by 'customer'?

> The DLF file contents are then imported into the
> database which is straighforward. Reports can then be  generated based on
> different timeslices and ocrrelations as desired.
> 
> This architecture is in harmony with the TelemetryBox which already has a
> SQL and Web capability. I would modify components on the telemetrybox to
> directly log in DLF format to the SQL database.

This would be very nice.

> Francis work would be useful from the web interface of the TelemetryBox so
> that reports can be generated from an ordinary user. The TelemetryBox can
> then also be configured to receive logfiles via email and work as a
> logprocessing node. Use feeds log data into the box and then either gets a
> report back via email or is able to remotely access the box and format
> the data according to his wishes.

Sounds good.

> Since the data is in a SQL database then also MS-Access (or other
> foreign frontends) can be also used via ODBC to query the data or extract
> data for furter processing as necessary.
> 
> Goals to be reached:
> 
> 1. The active user community would be increated by the availablility, easy
> installation and ease of use of the lire programm on nodes. We could
> generate a network of available monitoring nodes that process and forward
> data among one another.
> 
> 2. The develop community will be enhanced since it will be easy to adapt
> and integrate new reports and define new DLF structures.
> 
> 3. Instabillity: Integration into the TelemetryBox infrastructure would
> mean a package of open source tools that have been tested together is
> deployed. The issue of having the wrong version of tools does not arise.
> Deploy is very simple since the complete operating system is deployed. The
> deployment as an add-on onto existing operating systems (Sun, BSD etc) is
> still possible but carries with it the usual problems. The difficulty
> might increase since a SQL database and a webinterface might become
> necessary if we go down this route.

I would very much like to keep supporting Solaris, OpenBSD and FreeBSD.
Perhaps we could fully support a commandline interface on these
platforms, and support webfrontends only on GNU/Linux platforms.
I would very much regret is if we would only support Debian GNU/Linux.

> 4. Error messages: The SQL database does some error detection on import
> which would aid detection of errors. Special queries can be run to verify
> the integrity of the data. Conditions can be set in SQL to verify data.
> 
> 5. Report formats.
>    Reports are generated through embedded SQL in XML. The existing XML
>    converters can be used to generate a variety of output formats.
> 
> 6. Reconition of log files. Each type of log gets assigned a regex. Regex
>    for different log formats are tried until a match occurs then the
>   corresponding convertor is invoked.
> 
> What do you think about this?

Sounds ambitious and good.

Bye,

Joost



-- 
To UNSUBSCRIBE, email to development-request at logreport.org with a subject of 
"unsubscribe". Trouble? Send an email with subject "help" to 
development-request at logreport.org