WebMindä
Self-Organizing, Network-Based
Business Intelligence

WebMindä is a Java-based software system which evolves its own "digital intuition," and using this intuition, poses and answers questions regarding information. It deals with textual and numerical information on an equal footing, freely making generalizations spanning different types of data. Unlike any other AI system, it forms its own intuitions regarding concepts by dynamically "grounding" linguistic concepts in terms of its own nonlinguistic experience. It is the first AI system able to create its own concepts that apply across different contexts.

Derived from a sophisticated, complex-systems-based mathematical theory of intelligence, WebMindä's architecture is that of a massively parallel network, a population of many different static and dynamic agents continually recomputing their relationships with other agents, and acting on other agents in accordance with these relationships

WebMind Yields Dramatic Financial Intelligence Results

The Network is the Computer is the Mind

Knowledge Management and Knowledge Creation

Enterprise Analysis, Workflow and Business Process Reengineering

Webmind is a new kind of software system, but it incorporates elements or functions of software technologies typically used for business intelligence. It can serve as a search engine, as a Web visualization tool, or as a data mining engine -- and it can also serve functions not implied by any of these pre-existing categories. Most fundamentally, it is a system for the structuring and dynamic evolution of large bodies of data. It is an intelligent system for knowledge management and knowledge creation, with an almost boundless array of potential applications in business, science and personal computing.

Webmind collects and analyzes information in much the same way as the human mind does, but with different strengths and weaknesses, befitting its physical basis in silicon rather than carbon, and its perceptual embedding in the world of data and text feeds rather than the world of sights, smells, sounds, tastes and touches. In the human mind, everything above the lowest levels of perception and action is dynamic and constantly evolving. The memory constantly studies itself, reorganizing to enhance and inhibit recall and performance. Current information discovery tools impose an artificial distinction between static data repositories and dynamic search and analysis tools. But what is needed, in order to really make use of the vast amount of information we are amassing, is a dynamic information system that mimics, on an abstract level, the structure of the human mind, with all its reflexive, self-analytical capabilities. What is needed is a way to make the data come alive and study itself, rather than lying there inert and being probed by various computational instruments, of various degrees of intelligence.

This is what Webmind accomplishes. It allows collections of data to structure themselves and recognize patterns in themselves. Webmind transforms each informational element within a computer network into an independent agent which cooperates with other agents in organizing and reorganizing data for search and retrieval. It interacts with and builds on the intelligence of the user. Users' queries are treated as informational agents interacting via the collective Webmind agents system to locate information. Webmind agents take advantage of the sum collection of their organization's intelligence to discover context and concepts on behalf of users and their applications.

How does WebMind relate to the architectures and algorithms traditionally used in the AI field? The short answer is: it is completely different from all previous approaches, in that it is the first unified computational mind, complete with integrated systems for perception, action, cognition, language, short and long term memory, etc., all embedded in a data-rich environment.

The discipline of Artificial Intelligence is an extremely ambitious one, defined in the Artificial Intelligence Dictionary as "a multi-disciplinary field encompassing computer science, neuroscience, philosophy, psychology, robotics, and linguistics; and devoted to the reproduction of the methods or results of human reasoning and brain activity." The underlying long-range goal of AI is nothing less than the construction of thinking machines, machines that think as well as human beings, or better. In spite of these lofty ambitions, however, the reality of AI has up to this point been severely disappointing.

AI has not proved useless, but it has adopted a role as one technology among many, to be applied to tasks that

• Do not require a general, intuitive understanding of the world
• Are too non-routine for ordinary software methodologies
• Are too repetitive or too mathematically involved for humans to want to do, or to do well

As such, it has proved particularly useful in the finance industry, and in other specialized areas such as process control.

At the present time, the domain in which Webmind has been most systematically and scientifically tested is financial analytics. The goal in these tests was simple, to be able to make better financial predictions and trading decisions than one could without the software. The finance industry has already developed sophisticated tools for projecting trends in markets, using quantitative data about what the markets have done in the past. The problem with these techniques is that they do not encompass the full realm of information available to the markets.

The key advantage of Webmind for this purpose is that it can incorporate a logical analysis of textual material, material from financial news sources. This, of course, is very similar to what market analysts do. They study the statistical data, then interpret it in view of information which they obtain from their reading and conversations with industry leaders. However, there is a tremendous mass of written material available, and not enough time for any analyst to digest it all. Traders for large investment firms have difficulty even finding the time to read the reports generated by their own staff analysts. Webmind Analytics was designed to computerize this process.

The central purpose of Webmind Analytics is to make trading decisions. Webmind begins by incorporating the standard indicators and predictors which are already used in market analysis. An indicator provides a piece of information about the state of play at the time computed, such as the value of the market series at that time, the 5-session variability at that time, the Relative Strength Index of another data series at that time, or the incidence of meaningful text in internet news services at that time. We provide a list of standard and novel indicators which can be selected by the user from readily available data sources. A predictor takes in the past history of a market series, and perhaps other past information, and predicts the next value of the market series. We use standard statistical techniques such as linear and nonlinear regression, pattern searching and weighted historical combinations of indicators. All this is standard, state-of-the-art market analytics. What is new about Webmind is that textual indicators are analyzed and used together with the numerical indicators to make decisions about trading rules.

Initial tests on daily financial data proved dramatically successful. The following table shows our first results, indicating the percentage of correct prediction of market direction with and without the text-based indicators:

Dow Jones 63.5% 90.0%

Eurodollar futures 53.5% 81.3%

Yen futures 59.0% 83.3%

US T-bond futures 53.7% 68.2%

Cocoa futures 60.9% 82.5%

The correct predictions described in this table resulted in a high rate of return for Webmind's simulated trading system, as shown by the following annual profitability results on simulated trades over a year, based on a fixed drawdown and no reinvestment of profits:

Dow Jones 63.8% 172.3%

Eurodollar futures 22.9% 134.4%

Yen futures 35.4% 118.3%

US T-bond futures 7.8% 95.8%

Cocoa futures 19.9% 30.6%

One of the most exciting aspects of Webmind architecture is the way it leverages the intelligence intrinsic in the structure of modern computer networks. Its internal structure is that of a network of interconnected, meaning-sharing, learning agents; and using the power of Java for network programming, this internal network structure is made to harmonize with the network structure of modern intranets and the Internet.

Network-centric computing is commonly portrayed as a third phase in modern computing: first mainframes and terminals, then PC's and local-area networks ... and now large-scale, integrated network computing environments, providing the benefits of both mainframes and PC's and new benefits besides. In fact, though, this view underestimates the radical nature of the network-computing paradigm. What network computing truly represents is a return to the cybernetic, self-organization-oriented origins of computer science. It goes a long way toward correcting the fundamental error committed in the 1940's and 1950's, when the world decided to go with a serial, von-Neumann style computer architecture, to the almost total exclusion of more parallel, distributed, brain-like architectures.

The move to network computing is not merely a matter of evolving engineering solutions; it is also a matter of changing visions of computational intelligence. Mainframes and PC's mesh naturally with the symbolic, logic-based approach to intelligence; network computing environments, on the other hand, mesh with a view of the mind as a network of intercommunicating, intercreating processes. The important point is that the latter view of intelligence is the correct one. From computing frameworks supporting simplistic and fundamentally inadequate models of intelligence, one is suddenly moving to a computing framework supporting the real structures and dynamics of mind.

For mind and brain are fundamentally network-based. The mind, viewed system-theoretically, is far more like a network computing system than like a mainframe-based or PC-based system. It is not based on a central system that services dumb peripheral client systems, nor it is based on a huge host of small, independent, barely communicating systems. Instead it is a large, heterogeneous collection of systems, some of which service smart peripheral systems, all of which are intensely involved in inter-communication. In short, by moving to a network-computing framework, we are automatically supplying our computer systems with many elements of the structure and dynamics of mind.

Viewing computer networks in this way, one sees that network computing opens up fantastic new possibilities for artificial intelligence. Once one steps beyond the single-machine, single-program paradigm, and views the whole Net as a network of applets, able to be interconnected in various ways, it becomes clear that, in fact, every corporate Intranet -- and the Internet itself -- is an outstanding, terribly underutilized supercomputer. Each computer in a corporate intranet is potentially a "neuronal module" in

a company-wide electronic brain. Each computer hooked up to the Internet is potentially a part of the global brain; and each corporate intranet is potentially a lobe of a global business intelligence system.

networked, equipped with Java code or something similar, is potentially a "neuron" in a worldwide brain. The brain metaphor need not be taken too literally -- the point is that network computing, in principle, opens up the possibility for the Net to act as a dynamic, distributed cognitive system.

The structures of mind which are identified by the Psynet model as being most essential to intelligent functioning are:

A detailed description of the psynet model cannot be given here, but a capsule summary is as follows:

• A mind is a system of agents which are able to transform, create & destroy other agents
• Many of these agents act by recognizing patterns in the world, or in other agents
• Thoughts, feelings and other mental entities are self-reinforcing, self-producing systems of agents
• These self-producing mental subsystems build up into a complex network of attractors, meta-attractors, etc.
• This network of subsystems & associated attractors is "dual network" in structure, i.e. it is structured according to at least two principles: associativity (similarity) and hierarchy (categorization and category-based control).
• agents pass attention ("active force") to other agents to which they are related
• Because of finite memory capacity, mind must contain agents able to deal with "ungrounded" patterns, i.e. agents which were formed from now-forgotten agents, or which were learned from other minds rather than at first hand -- this is called "reasoning"
• A mind possesses agents whose goal is to recognize the mind as a whole as a pattern -- these are "self"

The Webmind AI system provides a general "agents operating system" for managing systems of software agents that share meaning amongst each other, transform each other, and interact in various ways; the agents may live in the RAM of a single machine, may be run on multiple processors, and may live across many machines connected by high-bandwidth cable. It provides mechanisms for agents to represent patterns they have recognized in other agents, and patterns they have recognized in the overall agents system of Webmind. And it also provides an appropriate assemblage of agents, specialized for such things as

• text processing
• numerical data processing
• text-numerical interrelation
• supervised and unsupervised concept formation
• reasoning
• social interaction
• language processing

These agents have their own localized behaviors but achieve their true intelligence only in the context of the whole Webmind system.

The psynet model of mind is too abstract to lead directly to any particular software program or engineering hardware design.. It can be implemented in many different ways. But, even without further specialization, it does say a great deal about AI. It dictates, for example,

• that an AI system must be a dynamical system, consisting of entities (agents) which are able to act on each other (transform each other) in a variety of ways, and some of which are able to evaluate simplicity (and hence recognize pattern).
• that this dynamical system must be sufficiently flexible to enable the crystallization of a dual network structure, with emergent, synergetic hierarchical and heterarchical subnets
• that this dynamical system must contain a mechanism for the spreading of attention in directions of shared meaning
• that this dynamical system must have access to a rich stream of perceptual data, so as to be able to build up a decent-sized pool of grounded patterns, leading ultimately to the recognition of the self
• that this dynamical system must contain entities that can reason (transfer information from grounded to ungrounded patterns)
• that this dynamical system must be contain entities that can manipulate categories (hierarchical subnets) and transformations involving categories in a sophisticated way, so as to enable syntax and semantics
• that this dynamical system must recognize symmetric, asymmetric and emergent meaning sharing, and build meanings using temporal and spatial relatedness, as well as relatedness of internal structure, and relatedness in the context of the system as a whole
• that this dynamical system must have a specific mechanism for paying extra attention to recently perceived data ("short-term memory")
• that this dynamical system must be embedded in a community of similar dynamical systems, so as to be able to properly understand itself

These criteria, while simple, are not met by any previously designed AI system.

Several questions arise in designing an AI implementation of the psynet model:

• 1. How to represent the agents
• 2. How to manage the dynamics of a collection of agents, on the hardware provided
• 3. What to build in, and what to require the system to learn
• 4. How to embed the system in a rich environment, including other systems of like kind
• 5. What goals to give the system
• 6. What programming language to use?

How one answers these questions determines what kind of system one comes up with. In this section we will describe how these questions were answered in the design of Webmind, on a general level.

In Webmind agents are realized as data structures called "nodes," with particular structural and dynamical properties. A psynet node is defined as a container for one or more of the following:

• possibly some "raw" perceptual data
• a record of the meaning of the current node, in the context of the mind as a whole
• a transformation which acts on a node, and produces a node
• a transformation which acts on a node, and produces a number
• tables of links to nodes with which it shares meaning in certain ways
• mobile agents which seek other nodes to add to the link tables
• information on how much attention is paid to the node currently and over the recent past, in total and from the perspective of various other nodes
• information about how important the node is, and how fast its importance is expected to decay over time

It should be clear that the node data structure, defined in this way, is adequate for expressing the "agent" concept of the psynet model of mind. Transformations taking agents into agents and agents into numbers are here, as required; and the dynamical law of mind, which says that attention follows meaning, is explicitly supported via each node carrying lists of other nodes that share its meaning, and agents searching for nodes that share its meaning.

The view of the mind as a network of psynet nodes is a kind of "middle way" for AI -- or in other words, an "intermediate level" model of intelligence. It stands between, on the one hand, neural network and other physiologically inspired models of intelligence; and on the other hand, abstract logical and psychological models of intelligence that speak of mind in the abstract, completely apart from its physical implementation. Neural network and logical and psychological models all have their particular validity, but considered as a collective, they leave a crucial gap: how do the complex, self-organizing physical dynamics of neural networks and the like give rise to the abstractions of the mind, such as syntax, logic, inference and self? "Psynet" means "mind network," and so it is only appropriate that the network interpretation of the psynet model fills in this gap by positing a network that is not a network of neurons but a network of intermediate-scale, mental entities ... "agent" nodes.

In a neurophysiological context, it may be useful to think of a psynet node as a neuronal group in the sense of Edelman's Neural Darwinism brain theory: each of Edelman's neuronal groups carries out a specific useful function, quite different from a single neuron, which in most cases has no particular function in itself. In some computer implementations of the psynet model, it may be useful to take this mapping fairly literally, and implement nodes in terms of networks of formal neurons -- as we shall see, something like this is done in Webmind, in certain cases, though not in general. But this is not a necessary approach: psynet nodes could be implemented as LISP program trees, as high-degree polynomials, as physical entities outside the realm of digital computing (Bose-Einstein condensates?), etc. The psynet model is abstract: it has to do with the structure and dynamics of the network of agents, not with the internal makeup of the agents themselves. Mind design, in this perspective, is largely about creating the correct agents out of the tools at hand, to ensure the emergence of the desired structures and dynamics from the system as a whole.

The dynamics of the network of nodes in Webmind's psynet is as follows. When a node receives attention, it spreads some of its attention to the meaning-sharing nodes that it knows about -- i.e. attention follows the links in its link table. It may also update its record of its own meaning. Finally, it may carry out its transformations to produce new nodes, or it may modify its transformations, with the goal of making them more prominent patterns in the collection of other nodes with which it shares meaning.

Ideally, each node would run independently on its own processor. However, that is not feasible at the present time: massively parallel processors do exist, but are orders of magnitude less efficient than networked top-of-the-line serial computers. Fortunately we are presently blessed with computers that, though serial in nature, possess multiple rapid processors, large amounts of RAM (in the gigabyte and up range), and fast cables (e.g. Ethernet) networking them together. This combination allows us to run large magician systems on serial machines.

The system architecture of Webmind is roughly as follows. A single mind network, may run across a number of machines, interconnected by high-bandwidth connections. Nodes may send messages across machines, and may share meaning with nodes on other machines; they may also move from one machine to another if this is judged dynamically efficient. Each machine contains a managerial process which determines when dispatching of nodes and messages between itself and other machines is appropriate. This managerial process must pay attention to the psynet dynamics on its own machine, for various reasons.

Within a single machine, nodes are divided into groups called pods. Grouping into pods is useful for human saving/loading of groups of nodes -- a pod of nodes may serve a particular human purpose, e.g. it may contain nodes concerning personnel records, nodes concerning financial data, etc. It is also useful for multiprocessing: a pod must live on a single processor, but multiple pods living on multiple processors may be very elegantly and efficiently handled. When a node wants to transform or spread attention to another node, living in a different pod, it sends a request to its pod's managerial process, which then sends the request on to the managerial process of the pod at the receiving end.

During the early development of Webmind, this was the toughest question to answer. One could seek to implement the psynet model of mind by setting up a kind of "primordial soup" of transformations, mapping into numbers and mapping into other transformations. Assuming this soup was hooked up to rich data feeds, given extra resources in return for success at practical tasks, and given a reasonable capacity to mutate itself, it would eventually transform itself into something intelligent, arriving at the structures described above by self-organizing evolution. This methodology, however, is extremely inefficient, and is extremely unlikely to work given the limitations of current engineering technology (it was tried in 1994 and failed miserably!). It mixes up what in biology are three different levels of intelligence: evolution, epigenesis, and learning. In order to arrive at a practical implementation of the psynet model, one must build in more of the desired structure -- just as a human being is born with some of the desired structure of mind already present in the structure of the brain.

The question then becomes how much one can build in without sacrificing the dynamicity, creativity, and self-modification that are at the essence of intelligence. The correct answer, we have found, is to engineer:

• the processing of low-level perception and action
• the types of agents potentially in the system (which determine not which mind you have, but what kind of mind you have)
• the specific configuration of agents, but only as a guide for future learning and not as a fixed, inorganic base of knowledge

The built-in preprocessing of low-level perception and action is necessary because of resource constraints. All that the human brain does by hard-wired visual circuitry -- line extraction, etc. -- could be done by more intelligent parts of the brain, but this would be a waste. The amount of pattern recognized in the world is maximized by allowing pre-processing to take care of the lowest level of pattern-recognition, and focusing the attention of intelligent on the output of the preprocessors. In the current Webmind system this lesson is reflected as follows: some basic text processing routines are built in, as are some numerical data processing routines that recognize dates in numerical data files, etc.

In the current system, a carefully chosen variety of node types is provided also: nodes for text, nodes for numerical data, nodes for words/phrases, nodes for senses of words, nodes for categories, etc. Each node type has specific means for determining shared meanings and for transforming itself and other nodes. The strategy is to build in the node types, and then allow the nodes, of selected types, to self-organize into their own structure based on their inputs. The key principle to remember here is the gradation of specialization. A system consisting of only highly specialized node types will never have the flexibility to be intelligent in the real world; but a system consisting of only highly general node types (not evolved or engineered for any particular world) will have a hard time coming to grips with its environment. Specialized node types are a step up from perceptual/active preprocessing, but serve a similar role: to enable the system to focus most of its attention on problems which require general intelligence.

In the current Webmind system, we have highly specialized node types for two things: language processing, and analysis of numerical data, particularly financial data (e.g. a node type specifically representing a trading system). Preprocessed data from these sensory modalities feeds into specialized nodes at the same time as it feeds into general-purpose nodes, and the interaction between the specialized and general-purpose nodes at this stage of processing is intense.

The interplay of different node types is is in some sense the hardest part of computational mind engineering. The essence of mind, as depicted by the psynet model, is relatively simple: what distinguishes one type of mind from another is mainly the types of agents that live inside it. The interaction between specialized agents and general-purpose agents is something that, in humans, evolved over millions of years; it can be engineered, but only with a firm understanding of the general underlying dynamics of mind, and a healthy respect for the difficulty of the task.

Humans receive a vast amount of perceptual data via their sense organs; and we interact with the same world producing this data via speaking, moving, stinking, etc. In short, we are embedded in our bodies. The body of Webmind consists of the computers on which it is implemented, which have their own sensory and active organs, for reading information from other computers and from human users. Pre-Internet, the data feeds available to computers were fairly sparse, and the ability for a computer program to interact with humans and other programs was strictly limited. At the present time, however, it is possible to hook up Webmind to a vast number of textual and numerical datafeeds, to supply with an unlimited amount of user interaction via corporate intranets and the Web, and to allow it to act by sending messages to individuals and other computers. For the first time, there is now a rich perceptual environment available to a computational intelligence. Furthermore, there is the opportunity to place a community of AI systems within this common environment, giving them access to common perceptual data and the ability to interact with one another in a variety of ways.

This is a somewhat obvious point, but it is one that has not been exploited by the mainstream of AI researchers, who do not accept the essentiality of a rich environment for the emergence of intelligence.

The social interaction of Webminds is explicitly mediated by data structures corresponding to the Self (the Webmind itself), other Webminds (needed for intelligent inter-Webmind querying), and human users (needed for intelligent customized response).

In order for a mind network to evolve effectively, it must be given goals to carry out, and must be given feedback on the quality with which it has carried out these goals. Just as humans are built to want to eat, have sex, drink, exercise, etc., so Webmind is built to want to answer queries effectively. In order to assure this, each time it sends a query, it should receive an assessment of the perceived quality of its response. This may be explicit (e.g., the user may be asked), or it may be implicit (e.g., the client may calculate the number of times the human user needed to rephrase his question to get the answer desired). We all know how hard it is to guess what humans want -- what this means is that answering human questions is certainly a "complex goal" as required in our working definition of intelligence, which is "the ability to achieve complex goals in complex environments."

The dynamics of the Psynet is fairly simple, though the emergent structures that these dynamics give rise to may be quite complex. Each pod carries its own threads, and these threads cycle through all the agents in the pod, allowing them to carry out "elemental actions." Each node carries a "heat" indicating its importance at a particular time, and hot nodes are given a greater priority.

Specialized pods for dealing with self, social interaction and creativity are dealt with in a fairly similar way to the PsyContext; they contain agents which, rather than being relevant to recent experience, are relevant to particular areas of moderately-recent experience, e.g. experience with oneself, experience with other intelligent entities, and experience with difficult problems.

The starting-point for Webmind's intelligence is a collection of basic node types used to represent data found on current computer networks. There are TextNodes for representing text documents, DataNodes for representing numerical data, and several types of CategoryNodes for representing concepts inferred from text and data. There are also TimeSets, representing collections of data that occur over time; and TimeNodes, representing collections of information considered roughly simultaneous. These node types are the basis for Webmind's intelligence; other processes work with these nodes to generate meaningful information of various forms. Information mined from data warehouses is read into the system and translated into a collection of TextNodes, DataNodes, CategoryNodes, TimeSets and TimeNodes; a flexible format which is amenable to sophisticated intelligent analysis by Webmind's various internal agents.

Parsing is carried out by a proprietary, original self-organizing parsing method based on L-system grammar. This system takes a sentence, uses the Psynet to determine syntactic and semantic categories for the words in the sentence, and then maps the sentence into a collection of nodes and relational agents, which are placed into the PsyContext and then into the Psynet. If the sentence represented a query, then the collection of nodes and agents corresponding to the sentence is activated rather than merely stored, and a query agent is created which collects the results of this activation after a period of time.

Mobile agents of various types create associative and hierarchical relationships, thus instantiating the elements of a cognitively powerful "dual network" structure. But what guarantees that these two structures, heterarchy and hierarchy, will work effectively together? There are several answers to this question within the Webmind architecture, but one of the keys is the explicit search for meaning. Each portion of the net, searching for its own meaning, re-expresses itself in terms of the hierarchical and heterarchical connections surrounding it, thus causing the hierarchical and heterarchical connections in its area to function effectively together.

Through processing of text alone, part of the fuzzy set that is the meaning of "two" can be acquired: that part which is purely linguistic, purely concerned with the appearance of "two" in linguistic combinations with other words. But the other part of the meaning of "two" can only be obtained by seeing "two" used, and using "two," in various situations, and observing the patterns in these situations that are correlated with correct use of "two." In general, the fuzzy set of patterns that is the meaning of a symbol X involves not only patterns emergent among X and other symbols, but also patterns emergent among X and entities involved in various situations. Recognition of patterns of this type is called symbol grounding.

The importance of reasoning can be well understood in the context of symbol grounding: reasoning allows Webmind to extend conclusions from grounded domains to ungrounded domains. For example, if Webmind has learned that "big" means a certain thing in the context of financial data (where it can ground the symbol "big" in reference to its direct experience), then it can use reasoning to transfer the relations it has derived about "bigness" to domains where it has no direct experience, e.g. to understand a phrase like "a big upheaval in the Japanese government."

When a query (be it a series of key words, a paragraph of natural language, or a series of commands requesting particular data operations) is entered into the system, a node is created for it, the query node sends out mobile agents, and these agents create new relational agents joining it and other nodes. Activity related to the query node spreads through the Psynet, and after a certain period of time, the nodes with the highest activity relevant to this particular activation process are collected, and returned as the answers to the query.

These levels may shift over time, as the Internet becomes faster and faster. On the other hand, analogy with biological intelligence leads us to distinguish two levels of systematic intelligence:

Intra-mental interactions may encompass social interactions, but not vice-versa: parts of the same mind have access to each other in ways that separate minds do not.

However, this mapping may not remain ideal forever.

Determining which server is best suited to receive a query or a visiting node to is done using a fairly sophisticated "socialization" system

Having reviewed the general nature of the Webmind architecture, we can now outline its application to various particular problem areas to a moderate level of detail. In this section we will consider several broad application domains: financial decision support, text information retrieval, data mining and enterprise analysis.

The application of Webmind to text-based market prediction described above requires a number of different node types. All of these node types have broad uses outside the domain of text-based market prediction, but are combined in this application in a special way. DataNodes were used for numerical data sets, such as financial time series. TextNodes are used for text documents, such as a market news report for a particular days. TimeSets are created to hold series of time-indexed nodes (TimeNodes), such as a series of market news reports over several years. ConceptNodes are created to hold concepts, which are either extracted from the text or learned by Webmind itself. TradingRuleNodes incorporated different financial trading rules which are routinely used by traders. TradingSystemNodes incorporated trading systems, which are collections of trading rules.

Suppose, for example, that the market being traded is the Dow Jones Industrial Average, and the text being used is market news which is issued on a daily basis. In this example, the data structures are a DataNode pointing to the DJIA, and a TimeSet pointing to the series of market news articles. The text indicator learning process is carried out by an agent associated with these two notes. The goal of this agent is to isolate a ConceptNode, representing one of Webmind's internal ideas, with the property that the relevance of the ConceptNode to the TimeNode, on a given day, is a useful indicator for trading the data in the DJIA DataNode on that day. Having carried out this learning process, it then uses the ConceptNode it has isolated to create a new "text indicator" DataNode, each entry of which indicates the relevance of the ConceptNode to the market news DynamicCategoryNode on a certain day. This text indicator DataNode can then by supplied to the user as, quite simply, a series of numbers. Each day, on reading the news, Webmind can supply a new value for the text indicator – just as, on seeing the DJIA itself, it can supply new values for numerical indicators such as moving averages and the like.

The text indicator obtained by the process of internal conceptualization can be used for a variety of different purposes. It can be used by humans to form their own intuitive judgments, or incorporated in any computational trading framework that is sufficiently flexible to incorporate arbitrary numerical indicator. Most of our work in this area, however, has involved incorporating text indicators into Webmind's own intelligent trading processes, which are designed to incorporate text indicator information. A TradingRuleNode, in Webmind, encapsulates a trading rule, which refers to a numerical predictor and a number of trading indicators, some numerical and some textual. A TradingSystemNode encapsulates a number of trading rules, one or more long rule and one or more short rule, which have been found to work well together. A genetic algorithm based learning technique, native to Webmind, is used to derive a collection of TradingRuleNodes which are near-optimal for a given DataNode, representing a certain financial instrument.

Here we will not go into these user interface matters in detail, but will rather give a cursory description of the internals of Webmind as applied to text information retrieval. This will serve to indicate how drastically different it is from its current competitors, and to illustrate the algorithmic roots of the different behavior that it displays. Each document being indexed is described by a TextNode, which contains links to various nodes representing the meaningful information in the text. There are also nodes corresponding to categories among documents. At the most abstract level, these categories might correspond to general topics such as "Business" or "Computing." At a level closer to the bottom, these might correspond to very particular groupings of pages, reflecting topics such as "artificial intelligence and finance." The process of category formation follows immediately from Webmind's self-organizing dynamics.

Each node on every level, when it is sufficiently stimulated, sends out a flock of intelligent agents, which wander the network of nodes in search of other nodes displaying similar or otherwise appropriately related patterns. When a query (be it a series of key words, a simple question, or a paragraph of natural language) is entered into the system, a node called a QueryNode is created for it. Then, the QueryNode is entered into the network much as if it were a new document to be read, classified and understood. First, using higher-level nodes to guide it, it finds a natural position in the network; then it sends out intelligent agents and these agents create new links between it and other nodes, other documents. Then activation spreads through the network, somewhat in the manner of a neural net, and after a certain period of time, the nodes with the highest activation are detected, and returned as the answers to the query.

What is fundamentally different about Webmind, as opposed to search engines, is that its document database is not a static entity, which sits there passively waiting for queries to set it into motion. Instead, it is constantly refining its structure -- and its structure is, like the structure of the human mind/brain, inseparable from its complex dynamics. The categories that it forms are not defined by mere statistical clustering, but by the outcome of self-organizing, reflexive dynamics. When a query is entered into Webmind, it sets off a chain of reactions that ultimately lead to it finding its natural places in the network, i.e., finding those groups of documents that pertain to the same topics as it does, as judged by the network's own complex dynamics.

In an ordinary search engine, a collection of pages is judged to have the "same topics" as each other if they have the same words in them. In a statistical search engine like Excite or SemioText, a collection of pages is judged to have a common topic if they have a mutually similar distribution of words and phrases. In Webmind, on the other hand, common topicality is not defined entirely in terms of structure but also in terms of process. A group of pages has a common topic if they have been judged as related by the self-organizing dynamics of the network. The distribution of words and phrases does play a role in this dynamics, as does the distribution of human-created links between phrases. But these factors only give the initial conditions for the dynamics. In the end, the distribution of links in Webmind is determined as much by the distribution of links in Webmind as by any specific identifiable factor. A query entered into Webmind is not only a request for information, but a perturbation of the complex dynamics of Webmind's constant study of itself.

It was noted above that Webmind can be used to form a more complete and complex version of the Yahoo category system. However, there are significant differences, which are conceptually relevant. The Yahoo category structure is very deep, for one thing; Webmind's structures tend to be somewhat flatter. Also, the Yahoo structure is strictly hierarchical, whereas, Webmind , like the human mind, is capable of operating with a patchwork of overlapping, complementary categorization schemes. In other words, the hierarchy of nodes can be dissociated, branching off in a number of different directions as it rises. Without Webmind's integrative dynamics, this could lead to confusion, but the dynamics are capable of integrating diverse points of view into a single coherent system -- in fact, this is what they do best. The different views within Webmind will not contradict but rather enhance each other. Ultimately, in a situation of extremely favorable computational resources, one can envision each user creating their own view of the Web, and allowing Webmind to crystallize a universal categorization system reflecting their individual perspective. The synthesis of all these individual views, according to Webmind dynamics, would be a kind of collective mentality -- not an average, but a complex, polyphonic blending of the different subjective world-views of the different members of the human race.

Webmind can use the same text-based interface as existing search engines. However, it also lends itself extremely naturally to visual, dynamic-image-directed search. Instead of returning lists of pages, a Webmind search engine can just as naturally return visual images of those portions of the Webmind network containing relevant pages. Different levels of magnification reveal lower and lower levels of the network, different colors represent complementary categorization schemes, etc. There are endless possibilities.

Next, regarding Webmind and the network on which it runs, it should be noted that, just as the general Webmind architecture is perfectly designed for network implementation, so is the Webmind search engine in particular. Today, it is assumed that a search engine database is a single, monolithic mirror of the Web. Already, however, some firms are breaking with this model; for example, FunnelWeb has constructed a network of search engines, one for each continent. When the Australian FunnelWeb engine wants information about the European FunnelWeb engine, it calls it up and asks it. This kind of networked search, on the level of passing queries from one engine to another, is relatively primitive and unexciting. However, Webmind opens up a whole new spectrum of possibilities.

Imagine, as the simplest case, that one set up two separate Webmind databases corresponding to .com and .edu domain names. Each one would have its own integrity, its own hierarchical/associative structure. But there could be links from one database to the other. And if agents from one database were allowed to wander through the other, then the two databases would be linked together on the level of dynamic structure: they would be two entrained, complex dynamical systems. The closeness of the link between the two systems would be determined by the propensity of the agents to leave their home database.

Imagine, next, that instead of two separate Webmind databases, one had, say, ten million. Imagine, in other words, that every website was its own Webmind database, and that, in order to do a search, one simply sent out intelligent agents through the Web. This is the opposite extreme from the one-search-engine/one-database model, and it is not yet feasible given current bandwidth limitations. But a search engine distributed across hundreds of sites is feasible today, and given the projected rapid growth of network bandwidth, the millions-of-sites scenario may well be feasible in ten years time. Once it is feasible, one will have something remarkable indeed: one will have the possibility for a World Wide Web which is, structurally and dynamically, a mirror of the human mind. Instead of a World Wide Web, we will have, so to speak, a World Wide Brain, a global network intelligence.

The term "data mining" refers to the process of searching through a large volume of data to discover interesting and useful information. Text information retrieval, as discussed above, may be considered a special case of data mining, in which the data involved is purely textual. Much of human intelligence is "data mining" -- we are concerned with searching through a vast amount of perceptual information, regarding the outside world and our own brains as well, and finding the interesting and useful information obscured wherein. In practice, though, the term "data mining" is generally used to refer to the elicitation of patterns from numerical data such as financial time series, or data stored in databases such as customer information; and it is these applications that will be the focus in the following comments. Most applications labeled "data mining" involve prediction of future trends, or categorization of data collected over an interval of time.

Data mining answers business questions that traditionally were too time-consuming to resolve, finding predictive or categorical information that experts may miss, either because it is too subtle and complicated, or because it lies outside their psychological expectations. It provides answers to questions such as, "Which clients are most likely to respond to my next promotional mailing, and why?" and "Which direction is the price of the Deutschmark likely to move over the next twenty-four hours?" and "What kind of person is hitting my Website?"

What differentiates Webmind from other data mining packages is, first of all, its ability to ferret out subtle patterns in data. It is able to pick up regularities that other, less sophisticated algorithms miss. Just as neural networks and genetic algorithms go beyond linear statistics, Webmind goes beyond neural networks and genetic algorithms. Furthermore, Webmind's sophisticated Internet implementation allows it to deal with large databases in a highly efficient way. Unlike other data analysis algorithms, Webmind is intrinsically designed to deal with tremendously large amounts of information, stored across distributed networks. As demonstrated in the financial experiments reported above, it is also able to analyze textual and numerical information in a unified way -- a fact of the utmost importance given the vast amounts of conceptually interrelated textual and numerical data on the contemporary business computer network.

On a more conceptual level, what differentiates Webmind is the fact that it approaches a body of data in roughly the same way that the human mind approaches the outside world. It takes the body of data into itself, just as perception brings the world into the mind, and it "brings the data to life," enabling the data to transform itself and study itself. The core, low-level pattern recognition routines used by Webmind are not necessarily different from those used by other data mining systems. The difference lies in the way the low-level recognized patterns are combined to form an holistic understanding of the entire body of data. This holistic understanding is embodied in Webmind's link structure, after a period of evolution, and it results in Webmind being able to recognize extremely subtle and sophisticated patterns in its input data, in an efficient way.

In practical terms, the application of Webmind to data mining is fairly straightforward. Data is read into Webmind and forms itself into a population of nodes; at which point the Webmind dynamics goes to work, bringing out the structure implicit in the data. Questions posed to the system are then resolved in terms of the information that Webmind has already discovered, on its own. If one's information is stored on an intranet, then absolutely no pre-processing is required, as Webmind is optimized to process distributed intranet data. If one's information is stored in a conventional database, or some other format, this is not a serious problem either, as the addition of simple additional software modules allows Webmind's to speak SQL and other relevant database languages.

Webmind carries out the two basic tasks of data mining -- prediction and categorization -- in a flexible and unified way. Categorization is taken care of by link update methods, as described above. Categories among, say, customers of a business, or stocks in the market, are found in the same way that the Webmind search engine isolates categories among Web pages. As for prediction, this is essentially a matter of recognizing patterns in one data set (representing, e.g., the past history of a financial market), and then extrapolating these patterns to another, perhaps unknown data set (e.g. the future of a financial market). The recognition of patterns is the crucial part. Neural networks and genetic algorithms pick up patterns that statistics and decision trees miss; and Webmind, with its self-organizing hierarchical/associative structure and its ability to read text and internally relate text with data, picks up patterns that neural networks and genetic algorithms miss.

As a single example of the application of Webmind to commercial data analysis, outside the finance industry, consider the problem of "targeted marketing." A business maintains a list of its customers over a period of time, which contains a fair amount of information about each customer. By studying this list, one can determine the qualities that typify a "good customer" for this business. The business then obtains a large database of names, e.g. census data, and wishes to determine which people from among this large database are likely to be good customers.

The specifics of this problem will vary from case to case, depending on the nature of the database. But the basic principles underlying the application of Webmind to this type of problem are quite general and flexible. Most simply, one may begin by building a Webmind database consisting of the elements of the large database. In the case of census data, each Webmind node might correspond to a single block, or even a single household. One then sets Webmind to work, building informative links between the nodes. Different links specify different types of relationship. For instance, there will be "geographical links," which link two nodes and are weighted by the distance between the blocks represented by the nodes (two nodes will be linked by a geographical link only if they are relatively close together). There may be links tied to particular parameters, e.g. "age structure" links or "income distribution" links. The different types of links are not chosen haphazardly, but are chosen based on the outcome of pattern recognition operations within the nodes. Each type of link is constructed by swarms of agents equipped with an appropriate criterion function, an appropriate measure of similarity between nodes. Higher level nodes are then formed, representing clusters amongst nodes -- natural groupings of "mutually similar" nodes. Finally, the elements of the smaller database are then entered into this network like queries into a text search oriented network. They find their most natural place in the network, extend links, and are activated. The most active nodes are then the "similar" nodes within the larger database.

This differs from typical statistical or AI approaches to market analysis in many essential respects. In terms of performance, the biggest difference is Webmind's ability to carry out highly targeted market analysis. Existing algorithms are effective at weeding out terrible customers from the large database, but only mediocre as isolating the best customers. Webmind, on the other hand, has a deeper understanding of the structure of the large database, because it begins by allowing the structure of the large database to discover itself. This deeper understanding allows it to target precisely those customers who are most similar to the business's best customers -- just as, in the Webmind search engine, a query finds those Web pages that are the most similar to it.

Similar approaches have been worked out for diverse other data mining applications -- no business intelligence problem lies outside its domain. The architecture encompasses the most effective aspects of previous approaches -- such as statistical clustering, neural networks, and genetic algorithms -- into an overarching framework. It integrates the two main objectives of data mining, categorization (carried out by higher-level nodes) and prediction (carried out by pattern-recognition methods within nodes), in a seamless way. As with Web search, it is not a panacea: there is still a certain amount of work required to tailor the prediction and link update methods to fit the nature of particular application domains. And searches for extremely subtle patterns in huge data warehouses can still take time, even with Webmind's sophisticated internetworking capabilities. But Webmind represents a significant step beyond what is out there; and it has the flexibility and reflexive adaptability to take advantage of improvements in Internet technology and data mining algorithms, without straying from its basic framework, which is, after all, nothing else but the structure of mind.

Collectively, these functions amount to making a map of an organization as a self-organizing system, and identifying the emergent structures in the organization. These functionalities will play an important role in enabling organizations to become more effective learning systems.